## Changes
This updates `mode: production` to allow `root_path` to indicate
uniqueness. Historically, we required `run_as` for this, which isn't
actually very effective for that purpose. `run_as` also had the problem
that it doesn't work for pipelines.
This is a cherry-pick from https://github.com/databricks/cli/pull/1387
---------
Co-authored-by: Pieter Noordhuis <pcnoordhuis@gmail.com>
Bumps [astral-sh/setup-uv](https://github.com/astral-sh/setup-uv) from 4
to 5.
<details>
<summary>Release notes</summary>
<p><em>Sourced from <a
href="https://github.com/astral-sh/setup-uv/releases">astral-sh/setup-uv's
releases</a>.</em></p>
<blockquote>
<h2>v5.0.0 🎄 Merry Christmas - Help fastly and users by default</h2>
<h2>Changes</h2>
<p>This christmans 🎄 release is a bit early bit still full of presents 🎁
Since we are changing some of the defaults this can lead to breaking
changes, thus the major version increase.</p>
<p>Here are the highlights:</p>
<h3><a
href="https://redirect.github.com/astral-sh/setup-uv/pull/193">Default
to enable-cache: true on GitHub hosted runners</a></h3>
<p>Did you know that that Fastly, the company hosting PyPI,
theoretically has to pay $12.5 million per month and so far have served
more than 2.41 <strong>exabytes</strong> of data?
<img
src="https://github.com/user-attachments/assets/f2f6cb3f-68f6-4e37-abb1-d3bf1f278533"
alt="image" /></p>
<p>This is why <a
href="https://redirect.github.com/astral-sh/setup-uv/issues/54">they
asked us</a> to turn on caching by default. After weighting the pros and
cons we decided to automatically upload the cache to the GitHub Actions
cache when running on GitHub hosted runners. You can still disable that
with <code>enable-cache: false</code>.</p>
<p>I remember when I first got into actions and didn't understand all
the magic. I was baffled that some actions did something behind the
scenes to make everything faster. I hope with this change we help a lot
of users who are don't want to or are afraid to understand what
<code>enable-cache</code> does.</p>
<h3><a
href="https://redirect.github.com/astral-sh/setup-uv/pull/185">Add
**/requirements*.txt to default cache-dependency-glob</a></h3>
<p>If caching is enabled we automatically searched for a
<code>uv.lock</code> file and when this changed we knew we had to
refresh the cache. A lot of projects don't use this but rather the good
old <code>requirements.txt</code>. We now automatically search for both
<code>uv.lock</code>and <code>requirements*.txt</code> (this means also
<code>requirements-test.txt</code>, <code>requirements-dev.txt</code>,
...) files.
You can change this with <code>cache-dependency-glob</code></p>
<h3><a
href="https://redirect.github.com/astral-sh/setup-uv/pull/194">Auto
activate venv when python-version is set</a></h3>
<p>Some workflows install packages on the fly. This automatically works
when using a python version that is already present on the runner. But
if uv installs the version, e.g. because it is a free-threaded version
or an old one, it is a <a
href="https://astral.sh/blog/python-build-standalone">standalone-build</a>
and installing packages "into the system" is not possible.</p>
<p>We now automatically create a new virtual environment with <code>uv
venv</code> and activate it for the rest of the workflow if
<code>python-version</code> is used. This means you can now do</p>
<pre lang="yaml"><code>- name: Install uv
uses: astral-sh/setup-uv@auto-environment
with:
python-version: 3.13t
- run: uv pip install -i
https://pypi.anaconda.org/scientific-python-nightly-wheels/simple cython
</code></pre>
<h2>🚨 Breaking changes</h2>
<ul>
<li>Default to enable-cache: true on GitHub hosted runners <a
href="https://github.com/eifinger"><code>@eifinger</code></a> (<a
href="https://redirect.github.com/astral-sh/setup-uv/issues/193">#193</a>)</li>
<li>Add **/requirements*.txt to default cache-dependency-glob <a
href="https://github.com/eifinger"><code>@eifinger</code></a> (<a
href="https://redirect.github.com/astral-sh/setup-uv/issues/185">#185</a>)</li>
</ul>
<h2>🐛 Bug fixes</h2>
<ul>
<li>Always use api.github.com <a
href="https://github.com/eifinger"><code>@eifinger</code></a> (<a
href="https://redirect.github.com/astral-sh/setup-uv/issues/191">#191</a>)</li>
</ul>
<h2>🚀 Enhancements</h2>
<ul>
<li>Auto activate venv when python-version is set <a
href="https://github.com/eifinger"><code>@eifinger</code></a> (<a
href="https://redirect.github.com/astral-sh/setup-uv/issues/194">#194</a>)</li>
<li>Add python version to cache key <a
href="https://github.com/eifinger"><code>@eifinger</code></a> (<a
href="https://redirect.github.com/astral-sh/setup-uv/issues/187">#187</a>)</li>
</ul>
<!-- raw HTML omitted -->
</blockquote>
<p>... (truncated)</p>
</details>
<details>
<summary>Commits</summary>
<ul>
<li><a
href="887a942a15"><code>887a942</code></a>
Set VIRTUAL_ENV to .venv instead of .venv/bin (<a
href="https://redirect.github.com/astral-sh/setup-uv/issues/210">#210</a>)</li>
<li><a
href="d174a24c07"><code>d174a24</code></a>
Align use of <code>actions/setup-python</code> with uv docu (<a
href="https://redirect.github.com/astral-sh/setup-uv/issues/207">#207</a>)</li>
<li><a
href="12c852e6ba"><code>12c852e</code></a>
Remove uv version from cache key (<a
href="https://redirect.github.com/astral-sh/setup-uv/issues/206">#206</a>)</li>
<li><a
href="180f8b4439"><code>180f8b4</code></a>
Fix wrong cacheDependencyPathHash (<a
href="https://redirect.github.com/astral-sh/setup-uv/issues/201">#201</a>)</li>
<li><a
href="e3fb95a689"><code>e3fb95a</code></a>
Warn instead of fail for no-dependency-glob (<a
href="https://redirect.github.com/astral-sh/setup-uv/issues/200">#200</a>)</li>
<li><a
href="2af22b5b2d"><code>2af22b5</code></a>
chore: update known checksums for 0.5.11 (<a
href="https://redirect.github.com/astral-sh/setup-uv/issues/198">#198</a>)</li>
<li><a
href="dd578776bb"><code>dd57877</code></a>
Auto activate venv when python-version is set (<a
href="https://redirect.github.com/astral-sh/setup-uv/issues/194">#194</a>)</li>
<li><a
href="85aa0bf0c1"><code>85aa0bf</code></a>
chore: update known checksums for 0.5.10 (<a
href="https://redirect.github.com/astral-sh/setup-uv/issues/196">#196</a>)</li>
<li><a
href="1f2cbfa7bb"><code>1f2cbfa</code></a>
Bump <code>@types/node</code> from 22.10.1 to 22.10.2 (<a
href="https://redirect.github.com/astral-sh/setup-uv/issues/189">#189</a>)</li>
<li><a
href="25b3ce6330"><code>25b3ce6</code></a>
chore: update known checksums for 0.5.9 (<a
href="https://redirect.github.com/astral-sh/setup-uv/issues/195">#195</a>)</li>
<li>Additional commits viewable in <a
href="https://github.com/astral-sh/setup-uv/compare/v4...v5">compare
view</a></li>
</ul>
</details>
<br />
[](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores)
Dependabot will resolve any conflicts with this PR as long as you don't
alter it yourself. You can also trigger a rebase manually by commenting
`@dependabot rebase`.
[//]: # (dependabot-automerge-start)
[//]: # (dependabot-automerge-end)
---
<details>
<summary>Dependabot commands and options</summary>
<br />
You can trigger Dependabot actions by commenting on this PR:
- `@dependabot rebase` will rebase this PR
- `@dependabot recreate` will recreate this PR, overwriting any edits
that have been made to it
- `@dependabot merge` will merge this PR after your CI passes on it
- `@dependabot squash and merge` will squash and merge this PR after
your CI passes on it
- `@dependabot cancel merge` will cancel a previously requested merge
and block automerging
- `@dependabot reopen` will reopen this PR if it is closed
- `@dependabot close` will close this PR and stop Dependabot recreating
it. You can achieve the same result by closing it manually
- `@dependabot show <dependency name> ignore conditions` will show all
of the ignore conditions of the specified dependency
- `@dependabot ignore this major version` will close this PR and stop
Dependabot creating any more for this major version (unless you reopen
the PR or upgrade to it yourself)
- `@dependabot ignore this minor version` will close this PR and stop
Dependabot creating any more for this minor version (unless you reopen
the PR or upgrade to it yourself)
- `@dependabot ignore this dependency` will close this PR and stop
Dependabot creating any more for this dependency (unless you reopen the
PR or upgrade to it yourself)
</details>
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
## Changes
The main CLI entry point used to be a global variable, and the global
state had to be cleaned up after every test run. This hasn't been the
case for a while, and instead, the CLI is initialized in a function
call. State accumulated by a single CLI "instance" can no longer leak
into other instances, so we no longer have to perform cleanup.
## Tests
Existing tests pass.
See Makefile for explanation on difference between 'make fmt' and 'make
lint'.
I also removed lint.sh. Original motivation was to use it in aider, but
it's not a good fit there, because aider passes filenames and it does
not work well with most golang linters which requires whole packages to
work.
Follow up to #2062, #2056, #2051.
Giving 7 days to react before closing is too aggressive, IMO. Changed it
to 30.
Also changed 'stale' label from 30d to 60d.
Also removed dry-run setting, it does not appear to do anything.
## Changes
Previously diagnostics were not seen in JSON output mode. This change
prints them to stderr.
This also fixes acceptance tests to preprocess all output with
s/execPath/$CLI/ not just output.txt.
## Tests
Existing acceptance tests. In one case I've added non-json command to
check that they match in output.
## Changes
This is useful to track telemetry associated with the templates and can
later be useful for functional usecases as well. Mlops stacks does the
same here: https://github.com/databricks/mlops-stacks/pull/185
## Tests
Existing tests.
## Changes
This workflow only worked if it was triggered on the tag to publish
itself. This means it is not possible to release a version if the
workflow configuration at that tag is broken (as is the case for
v0.238.0 because of #2105).
This change adds a "tag" input that can be set when manually triggering
the workflow.
## Tests
* Succesful run with this change:
https://github.com/databricks/cli/actions/runs/12689281843
* Pull request that the run created:
https://github.com/microsoft/winget-pkgs/pull/209220
## Changes
This action uses a token to access the release artifacts and, as such,
needs to execute on the runner that's on the allowlist.
Related PRs:
* #2098
* #2077
Since merge happens first, before variable resolution, the two jobs are
seen as different.
I also updated override/job_cluster/script to include more of the
output.
It's easier to remember and type and also validated and part of help:
```
% go test ./acceptance -updat 2>&1 | grep updat
flag provided but not defined: -updat
-update
```
## Changes
This used to work because the permission bits for built-in templates
were hardcoded to 0644 for files and 0755 for directories.
As of #1912 (and the PRs it depends on), built-in templates are no
longer pre-materialized to a temporary directory and read directly from
the embedded filesystem. This built-in filesystem returns 0444 as the
permission bits for the files it contains. These bits are carried over
to the destination filesystem.
This change updates template materialization to always set the owner's
write bit. It doesn't really make sense to write read-only files and
expect users to work with these files in a VCS (note: Git only stores
the executable bit).
The regression shipped as part of v0.235.0 and will be fixed as of
v0.238.0.
## Tests
Unit tests.
## Changes
- New kind of test is added - acceptance tests. See acceptance/README.md
for explanation.
- A few tests are converted to acceptance tests by moving databricks.yml
to acceptance/ and adding corresponding script files.
As these tests run against compiled binary and can capture full output
of the command, they can be useful to support major changes such as
refactoring internal logging / diagnostics or complex variable
interpolation.
These are currently run as part of 'make test' but the intention is to
run them as part of integration tests as well.
### Benefits
- Full binary is tested, exactly as users get it.
- We're not testing custom set of mutators like many existing tests.
- Not mocking anything, real SDK is used (although the HTTP endpoint is
not a real Databricks env).
- Easy to maintain: output can be updated automatically.
- Can easily set up external env, such as env vars, CLI args,
.databrickscfg location etc.
### Gaps
The tests currently share the test server and there is global place to
define handlers. We should have a way for tests to override / add new
handlers.
## Tests
I manually checked that output of new acceptance tests matches previous
asserts.
## Changes
- Remove TestMain from integration tests and related checks.
- This fixes "go test" caching for integration tests.
The test_main.go files were added in
https://github.com/databricks/cli/pull/2009 to make sure integration
tests are not run as part of go test ./.... We recommend running make
test to run tests, which includes the packages to test (and excludes
integration).
## Tests
To test that caching works I ran a test twice:
```
+ CLOUD_ENV=aws
+ go test --timeout 3h -v -run TestDefaultPython/3.9 ./integration/bundle/
…
PASS
ok github.com/databricks/cli/integration/bundle (cached)
```
## Changes
This reverts commit 31552852ff.
These workflows were disabled in #2085.
They should work again now that we're using self-hosted runners (see
#2077).
## Tests
(inline)
## Changes
Migrate workflows to Databricks-hosted GitHub Actions runners.
The GitHub-hosted runners can no longer be used because of security
hardening.
## Changes
The comment block appears on all PRs, even if the integration tests are
automatically triggered. This is quite noisy. This change limits those
comments to PRs from forks.
## Tests
Have to try by merging...
## Changes
This PR:
1. Incrementally improves the error messages shown to the user when the
volume they are referring to in `workspace.artifact_path` does not
exist.
2. Performs this validation in both `bundle validate` and `bundle
deploy` compared to before on just deployments.
3. It runs "fast" validations on `bundle deploy`, which earlier were
only run on `bundle validate`.
## Tests
Unit tests and manually. Also, existing integration tests provide
coverage (`TestUploadArtifactToVolumeNotYetDeployed`,
`TestUploadArtifactFileToVolumeThatDoesNotExist`)
Examples:
```
.venv➜ bundle-playground git:(master) ✗ cli bundle validate
Error: cannot access volume capital.whatever.my_volume: User does not have READ VOLUME on Volume 'capital.whatever.my_volume'.
at workspace.artifact_path
in databricks.yml:7:18
```
and
```
.venv➜ bundle-playground git:(master) ✗ cli bundle validate
Error: volume capital.whatever.foobar does not exist
at workspace.artifact_path
resources.volumes.foo
in databricks.yml:7:18
databricks.yml:12:7
You are using a volume in your artifact_path that is managed by
this bundle but which has not been deployed yet. Please first deploy
the volume using 'bundle deploy' and then switch over to using it in
the artifact_path.
```
## Changes
https://github.com/databricks/mlops-stacks/pull/187 broke mlops-stacks
deployments for non-UC projects. Snoozing the test until upstream is
fixed.
## Tests
The test is skipped on my local machine. CI run will verify it's also
skipped on Github Actions runners.
By default it stops after 3 issues of a given type, which gives false
impression and also unhelpful if you fixing it with aider.
1000 is almost like unlimited but not unlimited in case there is a bug
in a linter.
## Changes
This PR adds back debugging functionality that was lost during migration
to `internal.Main` as an entry point for integration tests.
The PR that caused the regression:
https://github.com/databricks/cli/pull/2009. Specifically the addition
of internal.Main as the entrypoint for all integration tests.
## Tests
Manually, by trying to debug a test.
This is useful on CI and locally for debugging and being able to
copy-paste command to tweak the options.
Removed redundant and imprecise messages like "✓ Running tests ...".
On main branch: ‘make test’ takes about 33s
On this branch: ‘make test’ takes about 2.7s
(all measurements are for hot cache)
What’s done (from highest impact to lowest):
- Remove -coverprofile= option - this option was disabling "go test"'s
built-in cache and also it took extra time to calculate the coverage
(extra 21s).
- Exclude ./integration/ folder, there are no unit tests there, but
having it included adds significant time. "go test"'s caching also does
not work there for me, due to TestMain() presence (extra 7.2s).
- Remove dependency on "make lint" - nice to have, but slow to re-check
the whole repo and should already be done by IDE (extra 2.5s).
- Remove dependency on "make vendor" — rarely needed; on CI it is
already executed separately (extra 1.1s).
The coverage option is still available under "make cover". Use "make
showcover" to show it.
I’ve also removed separate "make testonly". If you only want tests, run
"make test". If you want lint+test run "make lint test" etc.
I've also modified the test command, removed unnecessary -short, -v,
--raw-command.
## Changes
- Enable new linter: testifylint.
- Apply fixes with --fix.
- Fix remaining issues (mostly with aider).
There were 2 cases we --fix did the wrong thing - this seems to a be a
bug in linter: https://github.com/Antonboom/testifylint/issues/210
Nonetheless, I kept that check enabled, it seems useful, just need to be
fixed manually after autofix.
## Tests
Existing tests
## Changes
I noticed a diff in the schema in #2052.
This check should be performed automatically.
## Tests
This PR includes a commit that changes the schema to check that the
workflow actually fails.
## Changes
- Add t.Helper() in testcli-related helpers, this ensures that output is
attributed correctly to test case and not to the helper.
- Modify testlcli.Run() to run process in foreground. This is needed for
t.Helper to work.
- Extend a few assertions with message to help attribute it to proper
helper where needed.
## Tests
Manually reviewed test output.
Before:
```
+ go test --timeout 3h -v -run TestDefaultPython/3.9 ./integration/bundle/
=== RUN TestDefaultPython
=== RUN TestDefaultPython/3.9
workspace.go:26: aws
golden.go:14: run args: [bundle, init, default-python, --config-file, config.json]
runner.go:206: [databricks stderr]:
runner.go:206: [databricks stderr]: Welcome to the default Python template for Databricks Asset Bundles!
...
testdiff.go:23:
Error Trace: /Users/denis.bilenko/work/cli/libs/testdiff/testdiff.go:23
/Users/denis.bilenko/work/cli/libs/testdiff/golden.go:43
/Users/denis.bilenko/work/cli/internal/testcli/golden.go:23
/Users/denis.bilenko/work/cli/integration/bundle/init_default_python_test.go:92
/Users/denis.bilenko/work/cli/integration/bundle/init_default_python_test.go:45
...
```
After:
```
+ go test --timeout 3h -v -run TestDefaultPython/3.9 ./integration/bundle/
=== RUN TestDefaultPython
=== RUN TestDefaultPython/3.9
init_default_python_test.go:51: CLOUD_ENV=aws
init_default_python_test.go:92: args: bundle, init, default-python, --config-file, config.json
init_default_python_test.go:92: stderr:
init_default_python_test.go:92: stderr: Welcome to the default Python template for Databricks Asset Bundles!
...
init_default_python_test.go:92:
Error Trace: /Users/denis.bilenko/work/cli/libs/testdiff/testdiff.go:24
/Users/denis.bilenko/work/cli/libs/testdiff/golden.go:46
/Users/denis.bilenko/work/cli/internal/testcli/golden.go:23
/Users/denis.bilenko/work/cli/integration/bundle/init_default_python_test.go:92
/Users/denis.bilenko/work/cli/integration/bundle/init_default_python_test.go:45
...
```
## Changes
Fix cases where accumulated diagnostics are lost instead of being
propagated further. In some cases it's not possible, add a comment
there.
## Tests
Existing tests
Denis Bilenko
Lennart Kats (databricks)
Gleb Kanterov
dependabot[bot]
Pieter Noordhuis
shreyas-goenka
Ilya Kuznetsov
Andrew Nester