## Changes
Add ApplyPythonMutator, which will fork the Python subprocess and
process pipe bundle configuration through it.
It's enabled through `experimental` section, for example:
```yaml
experimental:
pydabs:
enable: true
venv_path: .venv
```
For now, it's limited to two phases in the mutator pipeline:
- `load`: adds new jobs
- `init`: adds new jobs, or modifies existing ones
It's enforced that no jobs are modified in `load` and not jobs are
deleted in `load/init`, because, otherwise, it will break existing
assumptions.
## Tests
Unit tests
## Changes
Previously, the functions `Get` and `Index` returned `dyn.NilValue` to
indicate that a map key or sequence index wasn't found. This is a valid
value, so we need to differentiate between actual absence and a real
`dyn.NilValue`. We do this with the zero value of a `dyn.Value` (also
captured in the constant `dyn.InvalidValue`).
## Tests
* Unit tests.
* Renamed `Get` and `Index` to find and update all call sites.
## Changes
This PR fixes the behaviour when variables were not overridden with
lookup value from targets if these variables had any default value set
in the default target.
Fixes#1449
## Tests
Added regression test
## Changes
Using dynamic values allows us to retain references like
`${resources.jobs...}` even when the type of field is not integer, eg:
`run_job_task`, or in general values that do not map to the Go types for
a field.
## Tests
Integration test
## Changes
1. Removes `DefaultMutatorsForTarget` which is no longer used anywhere
2. Makes SnapshotPath a private field. It's no longer needed by data
structures outside its package.
FYI, I also tried finding other instances of dead code but I could not
find anything else that was safe to remove. I used
https://go.dev/blog/deadcode to search for them, and the other instances
either implemented an interface, increased test coverage for some of our
other code paths or there was some other reason I could not remove them
(like autogenerated functions or used in tests).
Good sign our codebase is mostly clean (at least superficially).
## Changes
We set the origin URL as metadata in any jobs created by DABs. This PR
makes sure user credentials do not leak into the set metadata in the
job.
## Tests
Unit test
---------
Co-authored-by: Pieter Noordhuis <pieter.noordhuis@databricks.com>
## Changes
To run bundle deploy from DBR we use an abstraction over the workspace
import / export APIs to create a `filer.Filer` and abstract the file
system. Walking the file tree in such a filer is expensive and requires
multiple API calls. This PR remove the two duplicate file tree walks
that happen by caching the result.
Bumps [golang.org/x/oauth2](https://github.com/golang/oauth2) from
0.20.0 to 0.21.0.
<details>
<summary>Commits</summary>
<ul>
<li><a
href="5fd42413ed"><code>5fd4241</code></a>
google: update compute token refresh</li>
<li>See full diff in <a
href="https://github.com/golang/oauth2/compare/v0.20.0...v0.21.0">compare
view</a></li>
</ul>
</details>
<br />
[](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores)
Dependabot will resolve any conflicts with this PR as long as you don't
alter it yourself. You can also trigger a rebase manually by commenting
`@dependabot rebase`.
[//]: # (dependabot-automerge-start)
[//]: # (dependabot-automerge-end)
---
<details>
<summary>Dependabot commands and options</summary>
<br />
You can trigger Dependabot actions by commenting on this PR:
- `@dependabot rebase` will rebase this PR
- `@dependabot recreate` will recreate this PR, overwriting any edits
that have been made to it
- `@dependabot merge` will merge this PR after your CI passes on it
- `@dependabot squash and merge` will squash and merge this PR after
your CI passes on it
- `@dependabot cancel merge` will cancel a previously requested merge
and block automerging
- `@dependabot reopen` will reopen this PR if it is closed
- `@dependabot close` will close this PR and stop Dependabot recreating
it. You can achieve the same result by closing it manually
- `@dependabot show <dependency name> ignore conditions` will show all
of the ignore conditions of the specified dependency
- `@dependabot ignore this major version` will close this PR and stop
Dependabot creating any more for this major version (unless you reopen
the PR or upgrade to it yourself)
- `@dependabot ignore this minor version` will close this PR and stop
Dependabot creating any more for this minor version (unless you reopen
the PR or upgrade to it yourself)
- `@dependabot ignore this dependency` will close this PR and stop
Dependabot creating any more for this dependency (unless you reopen the
PR or upgrade to it yourself)
</details>
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Bumps [golang.org/x/mod](https://github.com/golang/mod) from 0.17.0 to
0.18.0.
<details>
<summary>Commits</summary>
<ul>
<li><a
href="c0bdc7bd01"><code>c0bdc7b</code></a>
modfile: add API for godebug lines</li>
<li><a
href="6686f41697"><code>6686f41</code></a>
module: add COM0 and LPT0 to badWindowsNames</li>
<li>See full diff in <a
href="https://github.com/golang/mod/compare/v0.17.0...v0.18.0">compare
view</a></li>
</ul>
</details>
<br />
[](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores)
Dependabot will resolve any conflicts with this PR as long as you don't
alter it yourself. You can also trigger a rebase manually by commenting
`@dependabot rebase`.
[//]: # (dependabot-automerge-start)
[//]: # (dependabot-automerge-end)
---
<details>
<summary>Dependabot commands and options</summary>
<br />
You can trigger Dependabot actions by commenting on this PR:
- `@dependabot rebase` will rebase this PR
- `@dependabot recreate` will recreate this PR, overwriting any edits
that have been made to it
- `@dependabot merge` will merge this PR after your CI passes on it
- `@dependabot squash and merge` will squash and merge this PR after
your CI passes on it
- `@dependabot cancel merge` will cancel a previously requested merge
and block automerging
- `@dependabot reopen` will reopen this PR if it is closed
- `@dependabot close` will close this PR and stop Dependabot recreating
it. You can achieve the same result by closing it manually
- `@dependabot show <dependency name> ignore conditions` will show all
of the ignore conditions of the specified dependency
- `@dependabot ignore this major version` will close this PR and stop
Dependabot creating any more for this major version (unless you reopen
the PR or upgrade to it yourself)
- `@dependabot ignore this minor version` will close this PR and stop
Dependabot creating any more for this minor version (unless you reopen
the PR or upgrade to it yourself)
- `@dependabot ignore this dependency` will close this PR and stop
Dependabot creating any more for this dependency (unless you reopen the
PR or upgrade to it yourself)
</details>
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Bumps [golang.org/x/term](https://github.com/golang/term) from 0.20.0 to
0.21.0.
<details>
<summary>Commits</summary>
<ul>
<li><a
href="5f0bb72315"><code>5f0bb72</code></a>
go.mod: update golang.org/x dependencies</li>
<li>See full diff in <a
href="https://github.com/golang/term/compare/v0.20.0...v0.21.0">compare
view</a></li>
</ul>
</details>
<br />
[](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores)
Dependabot will resolve any conflicts with this PR as long as you don't
alter it yourself. You can also trigger a rebase manually by commenting
`@dependabot rebase`.
[//]: # (dependabot-automerge-start)
[//]: # (dependabot-automerge-end)
---
<details>
<summary>Dependabot commands and options</summary>
<br />
You can trigger Dependabot actions by commenting on this PR:
- `@dependabot rebase` will rebase this PR
- `@dependabot recreate` will recreate this PR, overwriting any edits
that have been made to it
- `@dependabot merge` will merge this PR after your CI passes on it
- `@dependabot squash and merge` will squash and merge this PR after
your CI passes on it
- `@dependabot cancel merge` will cancel a previously requested merge
and block automerging
- `@dependabot reopen` will reopen this PR if it is closed
- `@dependabot close` will close this PR and stop Dependabot recreating
it. You can achieve the same result by closing it manually
- `@dependabot show <dependency name> ignore conditions` will show all
of the ignore conditions of the specified dependency
- `@dependabot ignore this major version` will close this PR and stop
Dependabot creating any more for this major version (unless you reopen
the PR or upgrade to it yourself)
- `@dependabot ignore this minor version` will close this PR and stop
Dependabot creating any more for this minor version (unless you reopen
the PR or upgrade to it yourself)
- `@dependabot ignore this dependency` will close this PR and stop
Dependabot creating any more for this dependency (unless you reopen the
PR or upgrade to it yourself)
</details>
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Bumps [golang.org/x/text](https://github.com/golang/text) from 0.15.0 to
0.16.0.
<details>
<summary>Commits</summary>
<ul>
<li><a
href="9c2f3a2135"><code>9c2f3a2</code></a>
cmd/gotext: fix segfault in extract & rewrite commands</li>
<li><a
href="59e1219a5f"><code>59e1219</code></a>
message: optimize lookupAndFormat function for better performance</li>
<li><a
href="a20a3e2496"><code>a20a3e2</code></a>
x/text: update x/tools for go/ssa range-over-func fix</li>
<li>See full diff in <a
href="https://github.com/golang/text/compare/v0.15.0...v0.16.0">compare
view</a></li>
</ul>
</details>
<br />
[](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores)
Dependabot will resolve any conflicts with this PR as long as you don't
alter it yourself. You can also trigger a rebase manually by commenting
`@dependabot rebase`.
[//]: # (dependabot-automerge-start)
[//]: # (dependabot-automerge-end)
---
<details>
<summary>Dependabot commands and options</summary>
<br />
You can trigger Dependabot actions by commenting on this PR:
- `@dependabot rebase` will rebase this PR
- `@dependabot recreate` will recreate this PR, overwriting any edits
that have been made to it
- `@dependabot merge` will merge this PR after your CI passes on it
- `@dependabot squash and merge` will squash and merge this PR after
your CI passes on it
- `@dependabot cancel merge` will cancel a previously requested merge
and block automerging
- `@dependabot reopen` will reopen this PR if it is closed
- `@dependabot close` will close this PR and stop Dependabot recreating
it. You can achieve the same result by closing it manually
- `@dependabot show <dependency name> ignore conditions` will show all
of the ignore conditions of the specified dependency
- `@dependabot ignore this major version` will close this PR and stop
Dependabot creating any more for this major version (unless you reopen
the PR or upgrade to it yourself)
- `@dependabot ignore this minor version` will close this PR and stop
Dependabot creating any more for this minor version (unless you reopen
the PR or upgrade to it yourself)
- `@dependabot ignore this dependency` will close this PR and stop
Dependabot creating any more for this dependency (unless you reopen the
PR or upgrade to it yourself)
</details>
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
## Changes
<!-- Summary of your changes that are easy to understand -->
Add support for `math/rand.Intn` to DAB templates.
## Tests
<!-- How is this tested? -->
Unit tests.
## Changes
This fixes a last-minute regression that snuck into
https://github.com/databricks/cli/pull/1463: unfortunately we need to
use `USE IDENTIFIER('schema')` to select a schema for now. In the future
we expect we can just use `USE SCHEMA 'schema'`.
## Changes
This worked fine if the notebooks are located in the filer's root and
didn't if they are nested in a directory.
This change adds test coverage and fixes the underlying issue.
## Tests
Ran integration test manually.
## Changes
This makes the dbt-sql and default-sql templates public.
These templates were previously not listed and marked "experimental"
since structured streaming tables were still in gated preview and would
result in weird error messages when a workspace wasn't enabled for the
preview.
This PR also incorporates some of the feedback and learnings for these
templates so far.
## Changes
From the [documentation](https://pkg.go.dev/os#IsNotExist) on the
functions in the `os` package:
> This function predates errors.Is. It only supports errors returned by
the os package.
> New code should use errors.Is(err, fs.ErrNotExist).
This issue surfaced while working on using a different `vfs.Path`
implementation that uses errors from the `fs` package. Calls to
`os.IsNotExist` didn't return true for errors that wrap
`fs.ErrNotExist`.
## Tests
n/a
## Changes
This change adds support for Lakehouse monitoring in bundles.
The associated resource type name is "quality monitor".
## Testing
Unit tests.
---------
Co-authored-by: Pieter Noordhuis <pcnoordhuis@gmail.com>
Co-authored-by: Pieter Noordhuis <pieter.noordhuis@databricks.com>
Co-authored-by: Arpit Jasapara <87999496+arpitjasa-db@users.noreply.github.com>
## Changes
This PR adds a filer that'll allow us to read notebooks from the WSFS
using their full paths (with the extension included). The filer relies
on the existing workspace filer (and consequently the workspace
import/export/list APIs).
Using this filer along with a virtual filesystem layer
(https://github.com/databricks/cli/pull/1452/files) will allow us to use
our custom implementation (which preserves the notebook extensions)
rather than the default mount available via DBR when the CLI is run from
DBR.
## Tests
Integration tests.
---------
Co-authored-by: Pieter Noordhuis <pieter.noordhuis@databricks.com>
## Changes
Introduce `libs/vfs` for an implementation of `fs.FS` and friends that
_includes_ the absolute path it is anchored to.
This is needed for:
1. Intercepting file operations to inject custom logic (e.g., logging,
access control).
2. Traversing directories to find specific leaf directories (e.g.,
`.git`).
3. Converting virtual paths to OS-native paths.
Options 2 and 3 are not possible with the standard `fs.FS` interface.
They are needed such that we can provide an instance to the sync package
and still detect the containing `.git` directory and convert paths to
native paths.
This change focuses on making the following packages use `vfs.Path`:
* libs/fileset
* libs/git
* libs/sync
All entries returned by `fileset.All` are now slash-separated. This has
2 consequences:
* The sync snapshot now always uses slash-separated paths
* We don't need to call `filepath.FromSlash` as much as we did
## Tests
* All unit tests pass
* All integration tests pass
* Manually confirmed that a deployment made on Windows by a previous
version of the CLI can be deployed by a new version of the CLI while
retaining the validity of the local sync snapshot as well as the remote
deployment state.
## Changes
The prior join call calls `filepath.Join` which returns a cleaned
result.
Path cleaning, in turn, calls `filepath.FromSlash`.
## Tests
* Unit tests.
## Changes
This PR also fixes empty values variable overrides using the --var flag.
Now, using `--var="my_variable="` will set the value of `my_variable` to
the empty string instead of ignoring the flag altogether.
## Tests
The change using a unit test. Manually verified the `--var` flag works
now.
Bumps
[github.com/hashicorp/terraform-exec](https://github.com/hashicorp/terraform-exec)
from 0.20.0 to 0.21.0.
<details>
<summary>Release notes</summary>
<p><em>Sourced from <a
href="https://github.com/hashicorp/terraform-exec/releases">github.com/hashicorp/terraform-exec's
releases</a>.</em></p>
<blockquote>
<h2>v0.21.0</h2>
<p>ENHANCEMENTS:</p>
<ul>
<li>tfexec: Add <code>-allow-deferral</code> to
<code>(Terraform).Apply()</code> and <code>(Terraform).Plan()</code>
methods (<a
href="https://redirect.github.com/hashicorp/terraform-exec/pull/447">#447</a>)</li>
</ul>
</blockquote>
</details>
<details>
<summary>Changelog</summary>
<p><em>Sourced from <a
href="https://github.com/hashicorp/terraform-exec/blob/main/CHANGELOG.md">github.com/hashicorp/terraform-exec's
changelog</a>.</em></p>
<blockquote>
<h1>0.21.0 (May 17, 2024)</h1>
<p>ENHANCEMENTS:</p>
<ul>
<li>tfexec: Add <code>-allow-deferral</code> to
<code>(Terraform).Apply()</code> and <code>(Terraform).Plan()</code>
methods (<a
href="https://redirect.github.com/hashicorp/terraform-exec/pull/447">#447</a>)</li>
</ul>
</blockquote>
</details>
<details>
<summary>Commits</summary>
<ul>
<li><a
href="b6ae175a49"><code>b6ae175</code></a>
v0.21.0 [skip ci]</li>
<li><a
href="67e92f47c0"><code>67e92f4</code></a>
build(deps): bump github.com/hashicorp/terraform-json from 0.22.0 to
0.22.1 (...</li>
<li><a
href="64df8d266b"><code>64df8d2</code></a>
build(deps): bump github.com/hashicorp/terraform-json from 0.21.0 to
0.22.0 (...</li>
<li><a
href="af05782eea"><code>af05782</code></a>
build(deps): Bump workflows to latest trusted versions (<a
href="https://redirect.github.com/hashicorp/terraform-exec/issues/450">#450</a>)</li>
<li><a
href="1df7d521ed"><code>1df7d52</code></a>
build(deps): bump golang.org/x/net from 0.22.0 to 0.23.0 (<a
href="https://redirect.github.com/hashicorp/terraform-exec/issues/444">#444</a>)</li>
<li><a
href="6ea72951a2"><code>6ea7295</code></a>
build(deps): bump hashicorp/setup-copywrite from 1.1.2 to 1.1.3 in the
github...</li>
<li><a
href="a9c9728194"><code>a9c9728</code></a>
tfexec: Add <code>-allow-deferral</code> experimental options to
<code>Plan</code> and <code>Apply</code> comm...</li>
<li><a
href="c07c6785b7"><code>c07c678</code></a>
Reenable Dependabot for internal GitHub actions (<a
href="https://redirect.github.com/hashicorp/terraform-exec/issues/455">#455</a>)</li>
<li><a
href="259b9e96e9"><code>259b9e9</code></a>
build(deps): bump github.com/hashicorp/hc-install from 0.6.3 to 0.6.4
(<a
href="https://redirect.github.com/hashicorp/terraform-exec/issues/443">#443</a>)</li>
<li><a
href="46360f17ab"><code>46360f1</code></a>
build(deps): bump github.com/zclconf/go-cty from 1.14.3 to 1.14.4 (<a
href="https://redirect.github.com/hashicorp/terraform-exec/issues/441">#441</a>)</li>
<li>Additional commits viewable in <a
href="https://github.com/hashicorp/terraform-exec/compare/v0.20.0...v0.21.0">compare
view</a></li>
</ul>
</details>
<br />
[](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores)
Dependabot will resolve any conflicts with this PR as long as you don't
alter it yourself. You can also trigger a rebase manually by commenting
`@dependabot rebase`.
[//]: # (dependabot-automerge-start)
[//]: # (dependabot-automerge-end)
---
<details>
<summary>Dependabot commands and options</summary>
<br />
You can trigger Dependabot actions by commenting on this PR:
- `@dependabot rebase` will rebase this PR
- `@dependabot recreate` will recreate this PR, overwriting any edits
that have been made to it
- `@dependabot merge` will merge this PR after your CI passes on it
- `@dependabot squash and merge` will squash and merge this PR after
your CI passes on it
- `@dependabot cancel merge` will cancel a previously requested merge
and block automerging
- `@dependabot reopen` will reopen this PR if it is closed
- `@dependabot close` will close this PR and stop Dependabot recreating
it. You can achieve the same result by closing it manually
- `@dependabot show <dependency name> ignore conditions` will show all
of the ignore conditions of the specified dependency
- `@dependabot ignore this major version` will close this PR and stop
Dependabot creating any more for this major version (unless you reopen
the PR or upgrade to it yourself)
- `@dependabot ignore this minor version` will close this PR and stop
Dependabot creating any more for this minor version (unless you reopen
the PR or upgrade to it yourself)
- `@dependabot ignore this dependency` will close this PR and stop
Dependabot creating any more for this dependency (unless you reopen the
PR or upgrade to it yourself)
</details>
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Bumps [github.com/fatih/color](https://github.com/fatih/color) from
1.16.0 to 1.17.0.
<details>
<summary>Release notes</summary>
<p><em>Sourced from <a
href="https://github.com/fatih/color/releases">github.com/fatih/color's
releases</a>.</em></p>
<blockquote>
<h2>v1.17.0</h2>
<h2>What's Changed</h2>
<ul>
<li>Fix multi-parameter println spacing by <a
href="https://github.com/klauspost"><code>@klauspost</code></a> in <a
href="https://redirect.github.com/fatih/color/pull/228">fatih/color#228</a></li>
<li>ci: update Go and Staticcheck versions by <a
href="https://github.com/fatih"><code>@fatih</code></a> in <a
href="https://redirect.github.com/fatih/color/pull/222">fatih/color#222</a></li>
<li>Bump golang.org/x/sys from 0.14.0 to 0.17.0 by <a
href="https://github.com/dependabot"><code>@dependabot</code></a> in <a
href="https://redirect.github.com/fatih/color/pull/221">fatih/color#221</a></li>
<li>Bump actions/setup-go from 4 to 5 by <a
href="https://github.com/dependabot"><code>@dependabot</code></a> in <a
href="https://redirect.github.com/fatih/color/pull/217">fatih/color#217</a></li>
<li>Bump golang.org/x/sys from 0.17.0 to 0.18.0 by <a
href="https://github.com/dependabot"><code>@dependabot</code></a> in <a
href="https://redirect.github.com/fatih/color/pull/224">fatih/color#224</a></li>
</ul>
<h2>New Contributors</h2>
<ul>
<li><a href="https://github.com/klauspost"><code>@klauspost</code></a>
made their first contribution in <a
href="https://redirect.github.com/fatih/color/pull/228">fatih/color#228</a></li>
</ul>
<p><strong>Full Changelog</strong>: <a
href="https://github.com/fatih/color/compare/v1.16.0...v1.17.0">https://github.com/fatih/color/compare/v1.16.0...v1.17.0</a></p>
</blockquote>
</details>
<details>
<summary>Commits</summary>
<ul>
<li><a
href="b6598b12a6"><code>b6598b1</code></a>
Merge pull request <a
href="https://redirect.github.com/fatih/color/issues/228">#228</a> from
klauspost/fix-println-issue-218</li>
<li><a
href="00b1811a3d"><code>00b1811</code></a>
Fix multi-parameter println spacing</li>
<li><a
href="04994a819d"><code>04994a8</code></a>
Merge pull request <a
href="https://redirect.github.com/fatih/color/issues/224">#224</a> from
fatih/dependabot/go_modules/golang.org/x/sys-0.18.0</li>
<li><a
href="7526cad07e"><code>7526cad</code></a>
Merge branch 'main' into
dependabot/go_modules/golang.org/x/sys-0.18.0</li>
<li><a
href="8d058ca54c"><code>8d058ca</code></a>
Merge pull request <a
href="https://redirect.github.com/fatih/color/issues/222">#222</a> from
fatih/ci-updates</li>
<li><a
href="2ac809fb73"><code>2ac809f</code></a>
Bump golang.org/x/sys from 0.17.0 to 0.18.0</li>
<li><a
href="51a7bbf7dc"><code>51a7bbf</code></a>
ci: update Go and Staticcheck versions</li>
<li><a
href="799c49c639"><code>799c49c</code></a>
Merge pull request <a
href="https://redirect.github.com/fatih/color/issues/217">#217</a> from
fatih/dependabot/github_actions/actions/setup-go-5</li>
<li><a
href="f8e0ec9c97"><code>f8e0ec9</code></a>
Merge branch 'main' into
dependabot/github_actions/actions/setup-go-5</li>
<li><a
href="298abd8b9e"><code>298abd8</code></a>
Merge pull request <a
href="https://redirect.github.com/fatih/color/issues/221">#221</a> from
fatih/dependabot/go_modules/golang.org/x/sys-0.17.0</li>
<li>Additional commits viewable in <a
href="https://github.com/fatih/color/compare/v1.16.0...v1.17.0">compare
view</a></li>
</ul>
</details>
<br />
[](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores)
Dependabot will resolve any conflicts with this PR as long as you don't
alter it yourself. You can also trigger a rebase manually by commenting
`@dependabot rebase`.
[//]: # (dependabot-automerge-start)
[//]: # (dependabot-automerge-end)
---
<details>
<summary>Dependabot commands and options</summary>
<br />
You can trigger Dependabot actions by commenting on this PR:
- `@dependabot rebase` will rebase this PR
- `@dependabot recreate` will recreate this PR, overwriting any edits
that have been made to it
- `@dependabot merge` will merge this PR after your CI passes on it
- `@dependabot squash and merge` will squash and merge this PR after
your CI passes on it
- `@dependabot cancel merge` will cancel a previously requested merge
and block automerging
- `@dependabot reopen` will reopen this PR if it is closed
- `@dependabot close` will close this PR and stop Dependabot recreating
it. You can achieve the same result by closing it manually
- `@dependabot show <dependency name> ignore conditions` will show all
of the ignore conditions of the specified dependency
- `@dependabot ignore this major version` will close this PR and stop
Dependabot creating any more for this major version (unless you reopen
the PR or upgrade to it yourself)
- `@dependabot ignore this minor version` will close this PR and stop
Dependabot creating any more for this minor version (unless you reopen
the PR or upgrade to it yourself)
- `@dependabot ignore this dependency` will close this PR and stop
Dependabot creating any more for this dependency (unless you reopen the
PR or upgrade to it yourself)
</details>
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
## Changes
If only key was defined for a job in YAML config, validate previously
failed with segfault.
This PR validates that jobs are correctly defined and returns an error
if not.
## Tests
Added regression test
## Changes
Add `merge.Override` transform. It allows the override one `dyn.Value`
with another, preserving source locations for parts of the sub-tree
where nothing has changed. This is different from merging, where values
are concatenated.
`OverrideVisitor` is visiting the changes during the override process
and allows to control of what changes are allowed or update the
effective value.
The primary use case is Python code updating bundle configuration.
During override, we update locations only for changed values. This
allows us to keep track of locations where values were initially defined
and used for error reporting. For instance, merging:
```yaml
resources: # location=left.yaml:0
jobs: # location=left.yaml:1
job_0: # location=left.yaml:2
name: "job_0" # location=left.yaml:3
```
with
```yaml
resources: # location=right.yaml:0
jobs: # location=right.yaml:1
job_0: # location=right.yaml:2
name: "job_0" # location=right.yaml:3
description: job 0 # location=right.yaml:4
job_1: # location=right.yaml:5
name: "job_1" # location=right.yaml:5
```
produces
```yaml
resources: # location=left.yaml:0
jobs: # location=left.yaml:1
job_0: # location=left.yaml:2
name: "job_0" # location=left.yaml:3
description: job 0 # location=right.yaml:4
job_1: # location=right.yaml:5
name: "job_1" # location=right.yaml:5
```
## Tests
Unit tests
## Changes
This is one step toward removing the `path.Paths` struct embedding from
resource types.
Going forward, we'll exclusively use the `dyn.Value` tree for location
information.
## Tests
Existing unit tests that cover path resolution with fallback behavior
pass.
## Changes
Currently, the help output of services in preview doesn't show any of
their commands because the commands themselves are hidden as well.
This change updates that behavior to not hide commands in preview if the
service itself is also in preview. This makes the help output of
services in preview actually usable.
## Tests
n/a
## Changes
Currently, there are a number of issues with the non-happy-path flows
for token refresh in the CLI.
If the token refresh fails, the raw error message is presented to the
user, as seen below. This message is very difficult for users to
interpret and doesn't give any clear direction on how to resolve this
issue.
```
Error: token refresh: Post "https://adb-<WSID>.azuredatabricks.net/oidc/v1/token": http 400: {"error":"invalid_request","error_description":"Refresh token is invalid"}
```
When logging in again, I've noticed that the timeout for logging in is
very short, only 45 seconds. If a user is using a password manager and
needs to login to that first, or needs to do MFA, 45 seconds may not be
enough time. to an account-level profile, it is quite frustrating for
users to need to re-enter account ID information when that information
is already stored in the user's `.databrickscfg` file.
This PR tackles these two issues. First, the presentation of error
messages from `databricks auth token` is improved substantially by
converting the `error` into a human-readable message. When the refresh
token is invalid, it will present a command for the user to run to
reauthenticate. If the token fetching failed for some other reason, that
reason will be presented in a nice way, providing front-line debugging
steps and ultimately redirecting users to file a ticket at this repo if
they can't resolve the issue themselves. After this PR, the new error
message is:
```
Error: a new access token could not be retrieved because the refresh token is invalid. To reauthenticate, run `.databricks/databricks auth login --host https://adb-<WSID>.azuredatabricks.net`
```
To improve the login flow, this PR modifies `databricks auth login` to
auto-complete the account ID from the profile when present.
Additionally, it increases the login timeout from 45 seconds to 1 hour
to give the user sufficient time to login as needed.
To test this change, I needed to refactor some components of the CLI
around profile management, the token cache, and the API client used to
fetch OAuth tokens. These are now settable in the context, and a
demonstration of how they can be set and used is found in
`auth_test.go`.
Separately, this also demonstrates a sort-of integration test of the CLI
by executing the Cobra command for `databricks auth token` from tests,
which may be useful for testing other end-to-end functionality in the
CLI. In particular, I believe this is necessary in order to set flag
values (like the `--profile` flag in this case) for use in testing.
## Tests
Unit tests cover the unhappy and happy paths using the mocked API
client, token cache, and profiler.
Manually tested
---------
Co-authored-by: Pieter Noordhuis <pieter.noordhuis@databricks.com>
I've updated the `deploy_then_remove_resources` test template in the
previous PR, but didn't notice that it was used in the destroy test too.
Now destroy test also checks deletion of jobs
Gleb Kanterov
Pieter Noordhuis
Lennart Kats (databricks)
shreyas-goenka
Andrew Nester
dependabot[bot]
Arpit Jasapara
Aravind Segu
Miles Yucht
Ilia Babanov