Commit Graph

672 Commits

Author SHA1 Message Date
shreyas-goenka 054df2b58b
Fix workspace import test (#844)
Windows and unix have different new line characters. Separating the
string assertions here to make the test pass.
2023-10-06 18:09:56 +00:00
shreyas-goenka 847b6f4bc3
Fix import export integration tests on windows (#842)
We should be using the path package here because they are paths in WSFS
2023-10-06 10:28:18 +00:00
shreyas-goenka 1e9dbcfa2a
Add `--file` flag to workspace export command (#794)
This PR:
1. Adds the `--file` flag to the workspace export command. This allows
you to specify a output file to write to.
2. Adds e2e integration tests for the workspace export command
2023-10-05 13:20:33 +00:00
shreyas-goenka caade735e3
Improve `workspace import` command by allowing references to local files for content (#793)
## Changes
This PR makes a few really important QOL improvements to the `workspace
import` command.

They are:
1. Adds the `--file` flag, which allows a user to specify a file to read
the content from.
2. Wraps the most common error first time users of this command will run
into with a helpful hint.
3. Minor changes to the command Use string changing `PATH` ->
`TARGET_PATH`


## Tests
Integration tests. The newly added integration tests that check the
--file flag works as expected for both `SOURCE` and `AUTO` format.

Skipped the other formats because the API behaviour for them is
straightforward.
2023-10-05 12:48:59 +00:00
Lennart Kats (databricks) 0894584132
Minor template tweaks (#832)
## Changes

Minor tweaks to the template.
2023-10-04 15:27:09 +00:00
Andrew Nester 2ce4f74f9c
Release v0.207.0 (#839)
CLI:
* Refactor change computation for sync
([#785](https://github.com/databricks/cli/pull/785)).

Bundles:
* Allow digits in the generated short name
([#820](https://github.com/databricks/cli/pull/820)).
* Emit an error when incompatible all purpose cluster used with Python
wheel tasks ([#823](https://github.com/databricks/cli/pull/823)).
* Use normalized short name for tag value in development mode
([#821](https://github.com/databricks/cli/pull/821)).
* Added `python.DetectInterpreters` and other utils
([#805](https://github.com/databricks/cli/pull/805)).
* Mark artifacts properties as optional
([#834](https://github.com/databricks/cli/pull/834)).
* Added support for glob patterns in pipeline libraries section
([#833](https://github.com/databricks/cli/pull/833)).

Internal:
* Run tests to verify backend tag validation behavior
([#814](https://github.com/databricks/cli/pull/814)).
* Library to validate and normalize cloud specific tags
([#819](https://github.com/databricks/cli/pull/819)).
* Added test to submit and run various Python tasks on multiple DBR
versions ([#806](https://github.com/databricks/cli/pull/806)).
* Create a release PR in setup-cli repo on tag push
([#827](https://github.com/databricks/cli/pull/827)).

API Changes:
* Changed `databricks account metastore-assignments list` command to
return .
* Changed `databricks jobs cancel-all-runs` command with new required
argument order.
 * Added `databricks account o-auth-published-apps` command group.
* Changed `databricks serving-endpoints query` command . New request
type is .
 * Added `databricks serving-endpoints patch` command.
 * Added `databricks credentials-manager` command group.
 * Added `databricks settings` command group.
 * Changed `databricks clean-rooms list` command to require request of .
* Changed `databricks statement-execution execute-statement` command
with new required argument order.

OpenAPI commit bcbf6e851e3d82fd910940910dd31c10c059746c (2023-10-02)
Dependency updates:
* Bump github.com/google/uuid from 1.3.0 to 1.3.1
([#825](https://github.com/databricks/cli/pull/825)).
* Updated Go SDK to 0.22.0
([#831](https://github.com/databricks/cli/pull/831)).
2023-10-04 14:34:59 +00:00
Andrew Nester 706393b64f
Create a release PR in setup-cli repo on tag push (#827)
## Changes
Create a release PR in setup-cli repo on tag push
2023-10-04 14:03:37 +00:00
Andrew Nester aa54a8665a
Added support for glob patterns in pipeline libraries section (#833)
## Changes
Now it's possible to specify glob pattern in pipeline libraries section
and DAB will add all matched files as libraries

```
  pipelines:
    dummy:
      name: " DLT with Python files"
      target: "dlt_python_files"
      libraries:
        - file:
            path: ./*.py
```

## Tests
Added unit test
2023-10-04 13:23:13 +00:00
Andrew Nester 9b6a847178
Mark artifacts properties as optional (#834)
## Changes
Mark artifacts properties as optional

Fixes #816
2023-10-03 13:59:28 +00:00
shreyas-goenka 40ae23bb33
Refactor change computation for sync (#785)
## Changes
This PR pays some tech debt by refactoring sync diff computation into
interfaces that are more robust.

Specifically:
1. Refactor the single diff computation function into a `SnapshotState`
class that computes the target state only based on the current local
files making it more robust and not carrying over state from previous
iterations.
2. Adds new validations for the sync state which make sure that the
invariants that downstream code expects are actually held true. This
prevents a class of issues where these invariants break and the
synchroniser behaves unexpectedly.

Note, this does not change the existing schema for the snapshot, only
the way the diff is computed, and thus is backwards compatible (ie does
not require a schema version bump).

## Tests
<!-- How is this tested? -->
2023-10-03 13:47:46 +00:00
Andrew Nester e1d1e95525
Updated Go SDK to 0.22.0 (#831)
## Changes
Updated Go SDK to 0.22.0
2023-10-03 11:46:16 +00:00
Andrew Nester 79e271f859
Added test to submit and run various Python tasks on multiple DBR versions (#806)
## Changes
These tests allow us to get information for execution context
(PYTHONPATH, CWD) for various Python tasks and different cluster setups.

Note: this test won't be executed automatically as part of nightly
builds since it requires RUN_PYTHON_TASKS_TEST env to be executed.

## Tests
Integration test run successfully.

---------

Co-authored-by: Pieter Noordhuis <pieter.noordhuis@databricks.com>
2023-10-03 11:18:55 +00:00
dependabot[bot] 452565cbd3
Bump github.com/google/uuid from 1.3.0 to 1.3.1 (#825)
Bumps [github.com/google/uuid](https://github.com/google/uuid) from
1.3.0 to 1.3.1.
<details>
<summary>Release notes</summary>
<p><em>Sourced from <a
href="https://github.com/google/uuid/releases">github.com/google/uuid's
releases</a>.</em></p>
<blockquote>
<h2>v1.3.1</h2>
<h2><a
href="https://github.com/google/uuid/compare/v1.3.0...v1.3.1">1.3.1</a>
(2023-08-18)</h2>
<h3>Bug Fixes</h3>
<ul>
<li>Use .EqualFold() to parse urn prefixed UUIDs (<a
href="https://redirect.github.com/google/uuid/issues/118">#118</a>) (<a
href="574e687494">574e687</a>)</li>
</ul>
</blockquote>
</details>
<details>
<summary>Changelog</summary>
<p><em>Sourced from <a
href="https://github.com/google/uuid/blob/master/CHANGELOG.md">github.com/google/uuid's
changelog</a>.</em></p>
<blockquote>
<h2><a
href="https://github.com/google/uuid/compare/v1.3.0...v1.3.1">1.3.1</a>
(2023-08-18)</h2>
<h3>Bug Fixes</h3>
<ul>
<li>Use .EqualFold() to parse urn prefixed UUIDs (<a
href="https://redirect.github.com/google/uuid/issues/118">#118</a>) (<a
href="574e687494">574e687</a>)</li>
</ul>
<h2>Changelog</h2>
</blockquote>
</details>
<details>
<summary>Commits</summary>
<ul>
<li><a
href="b3cae7c306"><code>b3cae7c</code></a>
chore(master): release 1.3.1 (<a
href="https://redirect.github.com/google/uuid/issues/127">#127</a>)</li>
<li><a
href="7b8f57cbcc"><code>7b8f57c</code></a>
fix(ci): switch to release-please app (<a
href="https://redirect.github.com/google/uuid/issues/126">#126</a>)</li>
<li><a
href="e69e468f7c"><code>e69e468</code></a>
chore(ci): configure release-please, update contrib (<a
href="https://redirect.github.com/google/uuid/issues/122">#122</a>)</li>
<li><a
href="97c970d2a5"><code>97c970d</code></a>
chore(ci): add apidiff check for API compatibility (<a
href="https://redirect.github.com/google/uuid/issues/123">#123</a>)</li>
<li><a
href="53f93a84e6"><code>53f93a8</code></a>
docs: change godoc URL in README (<a
href="https://redirect.github.com/google/uuid/issues/124">#124</a>)</li>
<li><a
href="0b416df5d0"><code>0b416df</code></a>
docs: update link to RFC 4122 (<a
href="https://redirect.github.com/google/uuid/issues/93">#93</a>)</li>
<li><a
href="75e1ac5ba6"><code>75e1ac5</code></a>
docs: shell format go tool command (<a
href="https://redirect.github.com/google/uuid/issues/111">#111</a>)</li>
<li><a
href="d71986992e"><code>d719869</code></a>
docs: fix typo node_js docs (<a
href="https://redirect.github.com/google/uuid/issues/117">#117</a>)</li>
<li><a
href="574e687494"><code>574e687</code></a>
fix: Use .EqualFold() to parse urn prefixed UUIDs (<a
href="https://redirect.github.com/google/uuid/issues/118">#118</a>)</li>
<li><a
href="c91929c8bf"><code>c91929c</code></a>
Merge pull request <a
href="https://redirect.github.com/google/uuid/issues/121">#121</a> from
noahdietz/setup-codeowners-and-actions</li>
<li>Additional commits viewable in <a
href="https://github.com/google/uuid/compare/v1.3.0...v1.3.1">compare
view</a></li>
</ul>
</details>
<br />


[![Dependabot compatibility
score](https://dependabot-badges.githubapp.com/badges/compatibility_score?dependency-name=github.com/google/uuid&package-manager=go_modules&previous-version=1.3.0&new-version=1.3.1)](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores)

Dependabot will resolve any conflicts with this PR as long as you don't
alter it yourself. You can also trigger a rebase manually by commenting
`@dependabot rebase`.

[//]: # (dependabot-automerge-start)
[//]: # (dependabot-automerge-end)

---

<details>
<summary>Dependabot commands and options</summary>
<br />

You can trigger Dependabot actions by commenting on this PR:
- `@dependabot rebase` will rebase this PR
- `@dependabot recreate` will recreate this PR, overwriting any edits
that have been made to it
- `@dependabot merge` will merge this PR after your CI passes on it
- `@dependabot squash and merge` will squash and merge this PR after
your CI passes on it
- `@dependabot cancel merge` will cancel a previously requested merge
and block automerging
- `@dependabot reopen` will reopen this PR if it is closed
- `@dependabot close` will close this PR and stop Dependabot recreating
it. You can achieve the same result by closing it manually
- `@dependabot show <dependency name> ignore conditions` will show all
of the ignore conditions of the specified dependency
- `@dependabot ignore this major version` will close this PR and stop
Dependabot creating any more for this major version (unless you reopen
the PR or upgrade to it yourself)
- `@dependabot ignore this minor version` will close this PR and stop
Dependabot creating any more for this minor version (unless you reopen
the PR or upgrade to it yourself)
- `@dependabot ignore this dependency` will close this PR and stop
Dependabot creating any more for this dependency (unless you reopen the
PR or upgrade to it yourself)


</details>

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2023-10-03 10:48:40 +00:00
Serge Smertin 7d0f170eee
Added `python.DetectInterpreters` and other utils (#805)
This PR adds a few utilities related to Python interpreter detection:

- `python.DetectInterpreters` to detect all Python versions available in
`$PATH` by executing every matched binary name with `--version` flag.
- `python.DetectVirtualEnvPath` to detect if there's any child virtual
environment in `src` directory
- `python.DetectExecutable` to detect if there's python3 installed
either by `which python3` command or by calling
`python.DetectInterpreters().AtLeast("v3.8")`

To be merged after https://github.com/databricks/cli/pull/804, as one of
the steps to get https://github.com/databricks/cli/pull/637 in, as
previously discussed.
2023-10-03 10:47:09 +00:00
Pieter Noordhuis f1b068cefe
Use normalized short name for tag value in development mode (#821)
## Changes

The jobs backend propagates job tags to the underlying cloud provider's
resources. As such, they need to match the constraints a cloud provider
places on tag values. The display name can contain anything. With this
change, we modify the tag value to equal the short name as used in the
name prefix.

Additionally, we leverage tag normalization as introduced in #819 to
make sure characters that aren't accepted are removed before using the
value as a tag value.

This is a new stab at #810 and should completely eliminate this class of
problems.

## Tests

Tests pass.
2023-10-02 06:58:51 +00:00
Andrew Nester 775251d0dc
Emit an error when incompatible all purpose cluster used with Python wheel tasks (#823)
## Changes
Follow up for https://github.com/databricks/cli/pull/807 to also
validate configuration if existing cluster id is used.

## Tests
Added unit tests
2023-09-29 12:19:05 +00:00
Pieter Noordhuis 4226c88e98
Library to validate and normalize cloud specific tags (#819)
## Changes

Prompted by the proposed fix for a tagging-related problem in #810, I
investigated how tag validation works. This turned out to be quite a bit
more complex than anticipated. Tags at the job level (or cluster level)
are passed through to the underlying compute infrastructure and as such
are tested against cloud-specific validation rules. GCP appears to be
the most restrictive. It would be disappointing to always restrict to
`\w+`, so this package implements validation and normalization rules for
each cloud. It can pick the right cloud to use using a Go SDK
configuration.

## Tests

Exhaustive unit tests. The regular expressions were pulled by #814.
2023-09-29 08:49:08 +00:00
Pieter Noordhuis 3685eb16f4
Run tests to verify backend tag validation behavior (#814)
## Changes

Validation rules on tags are different per cloud (they are passed
through to the underlying clusters and as such must comply with
cloud-specific validation rules). This change adds tests to confirm the
current behavior to ensure the normalization we can apply is in line
with how the backend behaves.

## Tests

The new integration tests pass (tested locally).
2023-09-29 08:38:06 +00:00
Pieter Noordhuis 30b4b8ce58
Allow digits in the generated short name (#820)
## Changes

Digits were previously replaced by `_`.

## Tests

Additional test cases with uncommon variations of email addresses.
2023-09-29 06:58:40 +00:00
Andrew Nester f31e8b446c
Release v0.206.0 (#811)
Bundles:
* Enable target overrides for pipeline clusters
([#792](https://github.com/databricks/cli/pull/792)).
* Add support for regex patterns in template schema
([#768](https://github.com/databricks/cli/pull/768)).
* Make the default `databricks bundle init` template more
self-explanatory ([#796](https://github.com/databricks/cli/pull/796)).
* Make a notebook wrapper for Python wheel tasks optional
([#797](https://github.com/databricks/cli/pull/797)).
* Added a warning when Python wheel wrapper needs to be used
([#807](https://github.com/databricks/cli/pull/807)).

Internal:
* Added `process.Background()` and `process.Forwarded()`
([#804](https://github.com/databricks/cli/pull/804)).

Dependency updates:
* Bump golang.org/x/term from 0.11.0 to 0.12.0
([#798](https://github.com/databricks/cli/pull/798)).
* Bump github.com/hashicorp/terraform-exec from 0.18.1 to 0.19.0
([#801](https://github.com/databricks/cli/pull/801)).
* Bump golang.org/x/oauth2 from 0.11.0 to 0.12.0
([#802](https://github.com/databricks/cli/pull/802)).
2023-09-27 10:57:34 +00:00
Serge Smertin 7171874db0
Added `process.Background()` and `process.Forwarded()` (#804)
## Changes
This PR adds higher-level wrappers for calling subprocesses. One of the
steps to get https://github.com/databricks/cli/pull/637 in, as
previously discussed.

The reason to add `process.Forwarded()` is to proxy Python's `input()`
calls from a child process seamlessly. Another use-case is plugging in
`less` as a pager for the list results.

## Tests
`make test`
2023-09-27 09:04:44 +00:00
Andrew Nester 3ee89c41da
Added a warning when Python wheel wrapper needs to be used (#807)
## Changes
Added a warning when Python wheel wrapper needs to be used

## Tests
Added unit tests + manual run with different bundle configurations
2023-09-27 08:26:59 +00:00
dependabot[bot] 99cc01c91c
Bump golang.org/x/oauth2 from 0.11.0 to 0.12.0 (#802)
Bumps [golang.org/x/oauth2](https://github.com/golang/oauth2) from
0.11.0 to 0.12.0.
<details>
<summary>Commits</summary>
<ul>
<li><a
href="07085280e4"><code>0708528</code></a>
go.mod: update golang.org/x dependencies</li>
<li><a
href="a835fc4358"><code>a835fc4</code></a>
oauth2: move global auth style cache to be per-Config</li>
<li>See full diff in <a
href="https://github.com/golang/oauth2/compare/v0.11.0...v0.12.0">compare
view</a></li>
</ul>
</details>
<br />


[![Dependabot compatibility
score](https://dependabot-badges.githubapp.com/badges/compatibility_score?dependency-name=golang.org/x/oauth2&package-manager=go_modules&previous-version=0.11.0&new-version=0.12.0)](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores)

Dependabot will resolve any conflicts with this PR as long as you don't
alter it yourself. You can also trigger a rebase manually by commenting
`@dependabot rebase`.

[//]: # (dependabot-automerge-start)
[//]: # (dependabot-automerge-end)

---

<details>
<summary>Dependabot commands and options</summary>
<br />

You can trigger Dependabot actions by commenting on this PR:
- `@dependabot rebase` will rebase this PR
- `@dependabot recreate` will recreate this PR, overwriting any edits
that have been made to it
- `@dependabot merge` will merge this PR after your CI passes on it
- `@dependabot squash and merge` will squash and merge this PR after
your CI passes on it
- `@dependabot cancel merge` will cancel a previously requested merge
and block automerging
- `@dependabot reopen` will reopen this PR if it is closed
- `@dependabot close` will close this PR and stop Dependabot recreating
it. You can achieve the same result by closing it manually
- `@dependabot show <dependency name> ignore conditions` will show all
of the ignore conditions of the specified dependency
- `@dependabot ignore this major version` will close this PR and stop
Dependabot creating any more for this major version (unless you reopen
the PR or upgrade to it yourself)
- `@dependabot ignore this minor version` will close this PR and stop
Dependabot creating any more for this minor version (unless you reopen
the PR or upgrade to it yourself)
- `@dependabot ignore this dependency` will close this PR and stop
Dependabot creating any more for this dependency (unless you reopen the
PR or upgrade to it yourself)


</details>

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2023-09-26 15:40:46 +00:00
dependabot[bot] 7cc8b4c17f
Bump github.com/hashicorp/terraform-exec from 0.18.1 to 0.19.0 (#801)
Bumps
[github.com/hashicorp/terraform-exec](https://github.com/hashicorp/terraform-exec)
from 0.18.1 to 0.19.0.
<details>
<summary>Release notes</summary>
<p><em>Sourced from <a
href="https://github.com/hashicorp/terraform-exec/releases">github.com/hashicorp/terraform-exec's
releases</a>.</em></p>
<blockquote>
<h2>v0.19.0</h2>
<p>ENHANCEMENTS:</p>
<ul>
<li>Add support for <code>terraform test</code> command (<a
href="https://redirect.github.com/hashicorp/terraform-exec/issues/398">#398</a>)</li>
<li>Add support for <code>-refresh-only</code> flag for
<code>Plan</code> and <code>Apply</code> methods. (<a
href="https://redirect.github.com/hashicorp/terraform-exec/issues/402">#402</a>)</li>
<li>Add support for <code>-destroy</code> flag for <code>Apply</code>
(<a
href="https://redirect.github.com/hashicorp/terraform-exec/issues/292">#292</a>)</li>
</ul>
<p>BUG FIXES:</p>
<ul>
<li>Fix bug in which the <code>TF_WORKSPACE</code> env var was set to an
empty string, instead of being unset as intended. (<a
href="https://redirect.github.com/hashicorp/terraform-exec/issues/388">#388</a>)</li>
</ul>
</blockquote>
</details>
<details>
<summary>Changelog</summary>
<p><em>Sourced from <a
href="https://github.com/hashicorp/terraform-exec/blob/main/CHANGELOG.md">github.com/hashicorp/terraform-exec's
changelog</a>.</em></p>
<blockquote>
<h1>0.19.0 (August 31, 2023)</h1>
<p>ENHANCEMENTS:</p>
<ul>
<li>Add support for <code>terraform test</code> command (<a
href="https://redirect.github.com/hashicorp/terraform-exec/issues/398">#398</a>)</li>
<li>Add support for <code>-refresh-only</code> flag for
<code>Plan</code> and <code>Apply</code> methods. (<a
href="https://redirect.github.com/hashicorp/terraform-exec/issues/402">#402</a>)</li>
<li>Add support for <code>-destroy</code> flag for <code>Apply</code>
(<a
href="https://redirect.github.com/hashicorp/terraform-exec/issues/292">#292</a>)</li>
</ul>
<p>BUG FIXES:</p>
<ul>
<li>Fix bug in which the <code>TF_WORKSPACE</code> env var was set to an
empty string, instead of being unset as intended. (<a
href="https://redirect.github.com/hashicorp/terraform-exec/issues/388">#388</a>)</li>
</ul>
</blockquote>
</details>
<details>
<summary>Commits</summary>
<ul>
<li><a
href="44a44d702b"><code>44a44d7</code></a>
v0.19.0 [skip ci]</li>
<li><a
href="b55ace7c3f"><code>b55ace7</code></a>
Bump github.com/hashicorp/hc-install from 0.5.2 to 0.6.0 (<a
href="https://redirect.github.com/hashicorp/terraform-exec/issues/407">#407</a>)</li>
<li><a
href="1581bc7ae3"><code>1581bc7</code></a>
Fix test file extensions in e2e tests (<a
href="https://redirect.github.com/hashicorp/terraform-exec/issues/408">#408</a>)</li>
<li><a
href="abef36d075"><code>abef36d</code></a>
Bump github.com/zclconf/go-cty from 1.13.3 to 1.14.0 (<a
href="https://redirect.github.com/hashicorp/terraform-exec/issues/406">#406</a>)</li>
<li><a
href="08caa9a099"><code>08caa9a</code></a>
Update CHANGELOG.md</li>
<li><a
href="f1b3c610bb"><code>f1b3c61</code></a>
Update CHANGELOG.md</li>
<li><a
href="ec5a394c63"><code>ec5a394</code></a>
feat: add refresh-only flag for plan and apply methods (<a
href="https://redirect.github.com/hashicorp/terraform-exec/issues/402">#402</a>)</li>
<li><a
href="2210f68c70"><code>2210f68</code></a>
apply: allow use of -destroy flag for compatible terraform versions (<a
href="https://redirect.github.com/hashicorp/terraform-exec/issues/292">#292</a>)</li>
<li><a
href="6a6a61a676"><code>6a6a61a</code></a>
Bump actions/checkout from 3.5.3 to 3.6.0 (<a
href="https://redirect.github.com/hashicorp/terraform-exec/issues/404">#404</a>)</li>
<li><a
href="cff1cb8939"><code>cff1cb8</code></a>
tfexec: Initial test command support (<a
href="https://redirect.github.com/hashicorp/terraform-exec/issues/400">#400</a>)</li>
<li>Additional commits viewable in <a
href="https://github.com/hashicorp/terraform-exec/compare/v0.18.1...v0.19.0">compare
view</a></li>
</ul>
</details>
<br />


[![Dependabot compatibility
score](https://dependabot-badges.githubapp.com/badges/compatibility_score?dependency-name=github.com/hashicorp/terraform-exec&package-manager=go_modules&previous-version=0.18.1&new-version=0.19.0)](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores)

Dependabot will resolve any conflicts with this PR as long as you don't
alter it yourself. You can also trigger a rebase manually by commenting
`@dependabot rebase`.

[//]: # (dependabot-automerge-start)
[//]: # (dependabot-automerge-end)

---

<details>
<summary>Dependabot commands and options</summary>
<br />

You can trigger Dependabot actions by commenting on this PR:
- `@dependabot rebase` will rebase this PR
- `@dependabot recreate` will recreate this PR, overwriting any edits
that have been made to it
- `@dependabot merge` will merge this PR after your CI passes on it
- `@dependabot squash and merge` will squash and merge this PR after
your CI passes on it
- `@dependabot cancel merge` will cancel a previously requested merge
and block automerging
- `@dependabot reopen` will reopen this PR if it is closed
- `@dependabot close` will close this PR and stop Dependabot recreating
it. You can achieve the same result by closing it manually
- `@dependabot show <dependency name> ignore conditions` will show all
of the ignore conditions of the specified dependency
- `@dependabot ignore this major version` will close this PR and stop
Dependabot creating any more for this major version (unless you reopen
the PR or upgrade to it yourself)
- `@dependabot ignore this minor version` will close this PR and stop
Dependabot creating any more for this minor version (unless you reopen
the PR or upgrade to it yourself)
- `@dependabot ignore this dependency` will close this PR and stop
Dependabot creating any more for this dependency (unless you reopen the
PR or upgrade to it yourself)


</details>

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2023-09-26 14:52:37 +00:00
Andrew Nester 0daa0022af
Make a notebook wrapper for Python wheel tasks optional (#797)
## Changes
Instead of always using notebook wrapper for Python wheel tasks, let's
make this an opt-in option.

Now by default Python wheel tasks will be deployed as is to Databricks
platform.
If notebook wrapper required (DBR < 13.1 or other configuration
differences), users can provide a following experimental setting

```
experimental:
  python_wheel_wrapper: true
```

Fixes #783,
https://github.com/databricks/databricks-asset-bundles-dais2023/issues/8

## Tests
Added unit tests.

Integration tests passed for both cases

```
    helpers.go:163: [databricks stdout]: Hello from my func
    helpers.go:163: [databricks stdout]: Got arguments:
    helpers.go:163: [databricks stdout]: ['my_test_code', 'one', 'two']
    ...
Bundle remote directory is ***/.bundle/ac05d5e8-ed4b-4e34-b3f2-afa73f62b021
Deleted snapshot file at /var/folders/nt/xjv68qzs45319w4k36dhpylc0000gp/T/TestAccPythonWheelTaskDeployAndRunWithWrapper3733431114/001/.databricks/bundle/default/sync-snapshots/cac1e02f3941a97b.json
Successfully deleted files!
--- PASS: TestAccPythonWheelTaskDeployAndRunWithWrapper (214.18s)
PASS
coverage: 93.5% of statements in ./...
ok      github.com/databricks/cli/internal/bundle       214.495s        coverage: 93.5% of statements in ./...

```

```
    helpers.go:163: [databricks stdout]: Hello from my func
    helpers.go:163: [databricks stdout]: Got arguments:
    helpers.go:163: [databricks stdout]: ['my_test_code', 'one', 'two']
    ...
Bundle remote directory is ***/.bundle/0ef67aaf-5960-4049-bf1d-dc9e29157421
Deleted snapshot file at /var/folders/nt/xjv68qzs45319w4k36dhpylc0000gp/T/TestAccPythonWheelTaskDeployAndRunWithoutWrapper2340216760/001/.databricks/bundle/default/sync-snapshots/edf0b322cee93b13.json
Successfully deleted files!
--- PASS: TestAccPythonWheelTaskDeployAndRunWithoutWrapper (192.36s)
PASS
coverage: 93.5% of statements in ./...
ok      github.com/databricks/cli/internal/bundle       195.130s        coverage: 93.5% of statements in ./...

```
2023-09-26 14:32:20 +00:00
dependabot[bot] e1b5912f59
Bump golang.org/x/term from 0.11.0 to 0.12.0 (#798)
Bumps [golang.org/x/term](https://github.com/golang/term) from 0.11.0 to
0.12.0.
<details>
<summary>Commits</summary>
<ul>
<li><a
href="f413282cd8"><code>f413282</code></a>
go.mod: update golang.org/x dependencies</li>
<li>See full diff in <a
href="https://github.com/golang/term/compare/v0.11.0...v0.12.0">compare
view</a></li>
</ul>
</details>
<br />


[![Dependabot compatibility
score](https://dependabot-badges.githubapp.com/badges/compatibility_score?dependency-name=golang.org/x/term&package-manager=go_modules&previous-version=0.11.0&new-version=0.12.0)](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores)

Dependabot will resolve any conflicts with this PR as long as you don't
alter it yourself. You can also trigger a rebase manually by commenting
`@dependabot rebase`.

[//]: # (dependabot-automerge-start)
[//]: # (dependabot-automerge-end)

---

<details>
<summary>Dependabot commands and options</summary>
<br />

You can trigger Dependabot actions by commenting on this PR:
- `@dependabot rebase` will rebase this PR
- `@dependabot recreate` will recreate this PR, overwriting any edits
that have been made to it
- `@dependabot merge` will merge this PR after your CI passes on it
- `@dependabot squash and merge` will squash and merge this PR after
your CI passes on it
- `@dependabot cancel merge` will cancel a previously requested merge
and block automerging
- `@dependabot reopen` will reopen this PR if it is closed
- `@dependabot close` will close this PR and stop Dependabot recreating
it. You can achieve the same result by closing it manually
- `@dependabot show <dependency name> ignore conditions` will show all
of the ignore conditions of the specified dependency
- `@dependabot ignore this major version` will close this PR and stop
Dependabot creating any more for this major version (unless you reopen
the PR or upgrade to it yourself)
- `@dependabot ignore this minor version` will close this PR and stop
Dependabot creating any more for this minor version (unless you reopen
the PR or upgrade to it yourself)
- `@dependabot ignore this dependency` will close this PR and stop
Dependabot creating any more for this dependency (unless you reopen the
PR or upgrade to it yourself)


</details>

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2023-09-26 14:27:02 +00:00
Lennart Kats (databricks) 0c1516c4ba
Make the default `databricks bundle init` template more self-explanatory (#796)
This makes the default-python template more self-explanatory and adds a
few other tweaks for a better out-of-the-box experience.
2023-09-26 09:12:34 +00:00
shreyas-goenka 757d5efe8d
Add support for regex patterns in template schema (#768)
## Changes
This PR introduces support for regex pattern validation in our custom
jsonschema validator. This allows us to fail early if a user enters an
invalid value for a field.

For example, now this is what initializing the default template looks
like with an invalid project name:
```
shreyas.goenka@THW32HFW6T bricks % cli bundle init
Template to use [default-python]: 
Unique name for this project [my_project]: (_*_)
Error: invalid value for project_name: (_*_). Must consist of letter and underscores only.
```

## Tests
New unit tests and manually.
2023-09-25 09:53:38 +00:00
Pieter Noordhuis ee30277119
Enable target overrides for pipeline clusters (#792)
## Changes

This is a follow-up to #658 and #779 for jobs.

This change applies label normalization the same way the backend does.

## Tests

Unit and config loading tests.
2023-09-21 19:21:20 +00:00
Andrew Nester c65e59751b
Release v0.205.2 (#791)
CLI:
* Prompt for profile only in interactive mode
([#788](https://github.com/databricks/cli/pull/788)).

Internal:
* Added setup Python action
([#789](https://github.com/databricks/cli/pull/789)).
2023-09-21 14:46:35 +00:00
Andrew Nester aa9c2a1eab
Prompt for profile only in interactive mode (#788)
## Changes
Do not prompt for profiles if not in interactive mode

## Tests
Running sample Go code

```
cmd := exec.Command("databricks", "auth", "login", "--host", "***")
out, err := cmd.CombinedOutput()
```
Before the change
```
Error: ^D

exit status 1
```

After
```
No error (empty output)
```
2023-09-21 12:38:45 +00:00
Andrew Nester 4a9dcd3231
Added setup Python action (#789)
## Changes
Added setup Python action

---------

Co-authored-by: Pieter Noordhuis <pieter.noordhuis@databricks.com>
2023-09-21 12:21:39 +00:00
Pieter Noordhuis 46996b884d
Release v0.205.1 (#787)
Bundles:
* Use enums for default python template
([#765](https://github.com/databricks/cli/pull/765)).
* Make bundle deploy work if no resources are defined
([#767](https://github.com/databricks/cli/pull/767)).
* Added support for experimental scripts section
([#632](https://github.com/databricks/cli/pull/632)).
* Error when unknown keys are encounters during template execution
([#766](https://github.com/databricks/cli/pull/766)).
* Fall back to full Git clone if shallow clone is not supported
([#775](https://github.com/databricks/cli/pull/775)).
* Enable environment overrides for job tasks
([#779](https://github.com/databricks/cli/pull/779)).
* Increase timeout waiting for job run to 1 day
([#786](https://github.com/databricks/cli/pull/786)).

Internal:
* Update Go SDK to v0.19.3 (unreleased)
([#778](https://github.com/databricks/cli/pull/778)).
2023-09-20 11:46:47 +00:00
Pieter Noordhuis 3a812a61e5
Increase timeout waiting for job run to 1 day (#786)
## Changes

It's not uncommon for job runs to take more than 2 hours. On the client
side, we should not stop waiting for a job to complete if it is
intentionally running for a long time. If a job isn't supposed to run
this long, the user can specify a run timeout in the job specification
itself.

## Tests

n/a
2023-09-19 19:54:24 +00:00
Andrew Nester 43e2eefc27
Enable environment overrides for job tasks (#779)
## Changes
Follow up for https://github.com/databricks/cli/pull/658

When a job definition has multiple job tasks using the same key, it's
considered invalid. Instead we should combine those definitions with the
same key into one. This is consistent with environment overrides. This
way, the override ends up in the original job tasks, and we've got a
clear way to put them all together.

## Tests
Added unit tests
2023-09-18 14:13:50 +00:00
Pieter Noordhuis b3b00fd226
Update Go SDK to v0.19.3 (unreleased) (#778)
## Changes

This bump includes:
* A fix for token refreshes on Azure
* A fix for retrying requests without a request body (e.g. GET)

Full comparison at
https://github.com/databricks/databricks-sdk-go/compare/v0.19.1...dacb7f4fc878.

## Tests

n/a
2023-09-15 14:54:23 +00:00
shreyas-goenka 2c58deb2c5
Fall back to full Git clone if shallow clone is not supported (#775)
## Changes
Git repos hosted over HTTP do not support shallow cloning. This PR adds
retry logic if we detect shallow cloning is not supported.

Note I saw the match string `dumb http transport does not support
shallow capabilities` being reported in for different hosts on the
internet, so this should work accross a large class of git servers.
Howerver, it's not strictly necessary to have the `--depth` flag so we
can remove it if this issue is reported again.

## Tests
Tested manually. `bundle init` successfully downloads the private HTTP
repo reported during by internal user.
2023-09-15 09:14:51 +00:00
shreyas-goenka 327ab0e598
Error when unknown keys are encounters during template execution (#766)
## Tests
New unit test and manually
2023-09-14 15:53:20 +00:00
Andrew Nester 953dcb4972
Added support for experimental scripts section (#632)
## Changes
Added support for experimental scripts section

It allows execution of arbitrary bash commands during certain bundle
lifecycle steps.

## Tests
Example of configuration

```yaml
bundle:
  name: wheel-task


workspace:
  host: ***

experimental:
  scripts:
    prebuild: |
      echo 'Prebuild 1'
      echo 'Prebuild 2'
    postbuild: "echo 'Postbuild 1' && echo 'Postbuild 2'" 
    predeploy: |
      echo 'Checking go version...'
      go version
    postdeploy: |
      echo 'Checking python version...'
      python --version

resources:
  jobs:
    test_job:
      name: "[${bundle.environment}] My Wheel Job"
      tasks:
        - task_key: TestTask
          existing_cluster_id: "***"
          python_wheel_task:
            package_name: "my_test_code"
            entry_point: "run"
          libraries:
          - whl: ./dist/*.whl
```

Output
```bash
andrew.nester@HFW9Y94129 wheel % databricks bundle deploy
artifacts.whl.AutoDetect: Detecting Python wheel project...
artifacts.whl.AutoDetect: Found Python wheel project at /Users/andrew.nester/dabs/wheel
'Prebuild 1'
'Prebuild 2'

artifacts.whl.Build(my_test_code): Building...
artifacts.whl.Build(my_test_code): Build succeeded
'Postbuild 1'
'Postbuild 2'

'Checking go version...'
go version go1.19.9 darwin/arm64

Starting upload of bundle files
Uploaded bundle files at /Users/andrew.nester@databricks.com/.bundle/wheel-task/default/files!

artifacts.Upload(my_test_code-0.0.0a0-py3-none-any.whl): Uploading...
artifacts.Upload(my_test_code-0.0.0a0-py3-none-any.whl): Upload succeeded
Starting resource deployment
Resource deployment completed!
'Checking python version...'
Python 2.7.18
```
2023-09-14 10:14:13 +00:00
shreyas-goenka fe32c46dc8
Make bundle deploy work if no resources are defined (#767)
## Changes

This PR sets "resource" to nil in the terraform representation if no
resources are defined in the bundle configuration. This solves two
problems:

1. Makes bundle deploy work without any resources specified. 
2. Previously if a `resources` block was removed after a deployment,
that would fail with an error. Now the resources would get destroyed as
expected.

Also removes `TerraformHasNoResources` which is no longer needed.

## Tests
New e2e tests.
2023-09-13 22:50:37 +00:00
shreyas-goenka be55310cc9
Use enums for default python template (#765)
## Changes
This PR changes schema to use the enum type for the default template
yes/no questions.

## Tests
Manually
2023-09-13 17:57:31 +00:00
Pieter Noordhuis 96d807fb85
Release v0.205.0 (#771)
This release marks the public preview phase of Databricks Asset Bundles.

For more information, please refer to our online documentation at
https://docs.databricks.com/en/dev-tools/bundles/.

CLI:
* Prompt once for a client profile
([#727](https://github.com/databricks/cli/pull/727)).

Bundles:
* Use clearer error message when no interpolation value is found.
([#764](https://github.com/databricks/cli/pull/764)).
* Use interactive prompt to select resource to run if not specified
([#762](https://github.com/databricks/cli/pull/762)).
* Add documentation link bundle command group description
([#770](https://github.com/databricks/cli/pull/770)).
2023-09-12 14:35:36 +00:00
shreyas-goenka 21ff71ceea
Add documentation link bundle command group description (#770)
Help output:
```
shreyas.goenka@THW32HFW6T ~ % cli bundle -h
Databricks Asset Bundles.
Documentation URL: https://docs.databricks.com/en/dev-tools/bundles.

Usage:
  databricks bundle [command]
```

---------

Co-authored-by: Pieter Noordhuis <pieter.noordhuis@databricks.com>
2023-09-12 13:38:43 +00:00
Pieter Noordhuis 3cb74e72a8
Run environment related tests in a pristine environment (#769)
## Changes

If the caller running the test has one or more environment variables
that are used in the test already set, they can interfere and make tests
fail.

## Tests

Ran tests in `./cmd/root` with Databricks related environment variables
set.
2023-09-12 13:28:53 +00:00
Pieter Noordhuis a2775f836f
Use interactive prompt to select resource to run if not specified (#762)
## Changes

Display an interactive prompt with a list of resources to run if one
isn't specified and the command is run interactively.

## Tests

Manually confirmed:
* The new prompt works
* Shell completion still works
* Specifying a key argument still works
2023-09-11 18:03:12 +00:00
Pieter Noordhuis 0cb05d1ded
Prompt once for a client profile (#727)
## Changes

The previous implementation ran the risk of infinite looping for the
account client due to a mismatch in determining what constitutes an
account client between the CLI and SDK (see
[here](83443bae8d/libs/databrickscfg/profiles.go (L61))
and
[here](0fdc5165e5/config/config.go (L160))).

Ultimately, this code must never infinite loop. If a user is prompted
and selects a profile that cannot be used, they should receive that
feedback immediately and try again, instead of being prompted again.

Related to #726.

## Tests
<!-- How is this tested? -->
2023-09-11 15:32:24 +00:00
shreyas-goenka 373f441eb2
Use clearer error message when no interpolation value is found. (#764)
## Changes
This PR makes the error message clearer for when interpolation fails. 

## Tests
Existing unit test and manually
2023-09-11 15:23:25 +00:00
Pieter Noordhuis 44726d6444
Release v0.204.1 (#763)
Bundles:
* Fix conversion of job parameters
([#744](https://github.com/databricks/cli/pull/744)).
* Add schema and config validation to jsonschema package
([#740](https://github.com/databricks/cli/pull/740)).
* Support Model Serving Endpoints in bundles
([#682](https://github.com/databricks/cli/pull/682)).
* Do not include empty output in job run output
([#749](https://github.com/databricks/cli/pull/749)).
* Fixed marking libraries from DBFS as remote
([#750](https://github.com/databricks/cli/pull/750)).
* Process only Python wheel tasks which have local libraries used
([#751](https://github.com/databricks/cli/pull/751)).
* Add enum support for bundle templates
([#668](https://github.com/databricks/cli/pull/668)).
* Apply Python wheel trampoline if workspace library is used
([#755](https://github.com/databricks/cli/pull/755)).
* List available targets when incorrect target passed
([#756](https://github.com/databricks/cli/pull/756)).
* Make bundle and sync fields optional
([#757](https://github.com/databricks/cli/pull/757)).
* Consolidate environment variable interaction
([#747](https://github.com/databricks/cli/pull/747)).

Internal:
* Update Go SDK to v0.19.1
([#759](https://github.com/databricks/cli/pull/759)).
2023-09-11 11:57:21 +00:00
shreyas-goenka ad84abf415
Fix temporary directory cleanup for init repository downloading (#760)
## Changes
This PR fixes a bug where the temp directory created to download the
template would not be cleaned up.

## Tests
Tested manually. The exact process is described in a comment below.
2023-09-11 10:22:05 +00:00
Lennart Kats (databricks) a4e94e1b36
Fix author in setup.py (#761)
Fix author in setup.py showing <no value>
2023-09-11 08:59:48 +00:00