## Changes
This PR also starts storing the DBR versions in the context.
Go patch file used:
```
@@
var x expression
@@
-dbr.MockRuntime(x, true)
+dbr.MockRuntime(x, dbr.Environment{IsDbr: true, Version: "15.4"})
@@
var x expression
@@
-dbr.MockRuntime(x, false)
+dbr.MockRuntime(x, dbr.Environment{})
```
ref: https://github.com/uber-go/gopatch
## Why
This localised all DBR version accesses to `libs/dbr`. Relevant comment:
https://github.com/databricks/cli/pull/2432#discussion_r1982878616
## Tests
Exiting tests are modified.
## Changes
- New library to patch existing whl with dynamic version. The version is
based on mtime of original wheel.
- New hidden command "selftest patchwhl" (for acceptance test).
## Why
Part of the project to deprecate dynamic version in setup.py template
(#1034). This would enable switching default-python to pyproject.toml
without dependency on setuptools-specific sections.
The dynamic version is needed to ensure that wheels are picked up by the
users code when doing development.
## Tests
- New unit tests that use uv to create envs on various versions and
install (and re-install) patched wheels there.
- New acceptance test that patches prebuilt whl and records the result
fully unzipped.
- New acceptance test helper setmtime.py to set mtime with nanosecond
precision cross-platform.
## Changes
The `schema` field implies the lifecycle of tables is no longer tied to
the lifecycle of the pipeline, as was the case with the `target` field.
More information about using the "catalog" and "schema" properties can
be found here:
https://docs.databricks.com/en/delta-live-tables/target-schema.html
## Tests
n/a
---------
Co-authored-by: Lennart Kats (databricks) <lennart.kats@databricks.com>
## Changes
- Do not modify or edit .gitignore in bundle root.
- Instead create .databricks/.gitignore with content set to "*"
## Why
Merging our changes into existing .gitignore is complicated and adding
.gitignore where it's not expected adds to the noise. Other tools also
use the approach in this PR (e.g. ruff creates .ruff_cache/.gitignore).
## Tests
- Modified templates/default-sql to capture this new file.
## Changes
- Added `service-principal-secrets` command
- Added `budget-policy-id` for apps
- `experiments.log-inputs` now requires `ID` parameter as an input
- Added `genie.get-space` command
- Added `providers.list-provider-share-assets` command
For the whole list of SDK changes see:
https://github.com/databricks/databricks-sdk-go/releases/tag/v0.59.0
## Why
VSCode for me resolves `script` as YAML files showing incorrect syntax
coloring. This setting forces VSCode to treat all files named `script`
as shell scripts leading to better syntax highlighting.
These files are common place in our acceptance tests.
## Tests
Manually.
There is no perf difference, it's fast in both cases:
```
~/work/cli % hyperfine 'ruff format -qn'
Benchmark 1: ruff format -qn
Time (mean ± σ): 22.5 ms ± 4.1 ms [User: 23.6 ms, System: 106.7 ms]
Range (min … max): 17.3 ms … 50.2 ms 52 runs
```
but it no longer produces .ruff_cache which can confuse acceptance test
runner
```
~/work/cli % find . -type d | grep ruff
./.ruff_cache
./.ruff_cache/0.8.3
./libs/notebook/testdata/.ruff_cache
./libs/notebook/testdata/.ruff_cache/0.8.3
./libs/sync/testdata/.ruff_cache
./libs/sync/testdata/.ruff_cache/0.8.3
./acceptance/bundle/templates/default-sql/.ruff_cache
./acceptance/bundle/templates/default-sql/.ruff_cache/0.8.3
```
## Changes
Switch to use GET workspaces-files/{name} instead of workspace/export
for state files.
## Why
`/api/2.0./workspaces-files/{name}` has a higher limit which allows to
export state files larger than 10 MBs (which is the current limit for
`workspace/export`). We don't use the same API for read in other places
and fully replacing existing Filer because it doesn't correct get the
file content for notebooks and returns "File Not Found" error instead.
## Tests
All existing tests pass
## Changes
This function provides all environment variables necessary to
authenticate the downstream applications to the same credentials as the
parent process.
It's used in https://github.com/databricks/cli/pull/2278 and will also
be useful for the `databricks bundle exec` command.
## Tests
Unit test.
## Changes
1. Changed `FindResourceByConfigKey` to return schema resources
2. Implemented the `Exists` method for schema resources
## Why
This PR adds support for schema resources in deployment operations,
enabling users to:
- Bind schemas using `databricks bundle deployment bind <myschema_key>
<schema_full_name>`
- Unbind schemas using `databricks bundle deployment unbind
<myschema_key>`
Where:
- `myschema_key` is a resource key defined in the bundle's .yml file
- `schema_full_name` references an existing schema in the Databricks
workspace
These capabilities allow for more flexible resource management of
schemas within bundles.
## Tests
Added a new integration test that tests bind and unbind methods together
with bundle deployment and destruction.
## Changes
1. remove t.Skip directive from TestAuthDescribeFailure integration test
2. remove checking the host in the test
## Why
1. This enables integration test that exercises `databricks auth
describe` command, which was previously throwing errors in the CI/CD
pipeline. For the nonexistent profile check to fail, we need to create a
new profile file for the CLI to check (otherwise the configuration skips
checking of the provided profile actually exists).
2. I removed the assertion for the host part because it is not relevant
for the test, and setting that up just required more code in the test
setup.
## Tests
The integration test is passing
## Changes
Added a warning when `config` section is used in apps
## Why
To avoid the confusion between using apps in DABs and outside of DABs,
we want to provide only one way of configuring apps runtime
configuration - by using `app.yml` file in the root of the app.
## Tests
Added acceptance tests
## Changes
Lock setuptools version to 75.8.2 (latest as of March 3, 2025)
## Why
As part of the tests `uv install` was installing latest version of
setuptools which led to all tests started to fail on Feb 25 when 75.8.1
setuptools version was released and which changed the naming of the
output built artifacts
https://setuptools.pypa.io/en/stable/history.html#v75-8-1
This change prevents us from breakages like this.
## Tests
Existing tests pass
## Changes
Same name libraries check only valid for local libraries. Local
libraries are only supported for Whl and Jar types. Hence we can
restrict matching pattern only to these libraries.
## Tests
Existing acceptance tests pass
## Changes
This PR adds the auth.EnvVars function, which is a list of all
environment variables that the SDK uses to read auth configuration.
This is useful for spawning subprocesses since you can unset all auth
environment variables to clean up the environment before configuring the
auth.
It's used in #2278 today and will also be useful for the `databricks
bundle exec` command.
## Tests
Unit test.
## Changes
Add mutex synchronization in cmdio logger Log() method.
## Why
Since we issue multiple calls to underlying writer, we should lock the
whole method, otherwise we can get broken messages. One that can be
easily reproduced today is
```
hyperfine -m 100 --show-output 'go test ./acceptance -run ^TestAccept$/^bundle$/^artifacts$/^whl_multiple$ -count=1'
...
-Uploading my_test_code-0.0.1-py3-none-any.whl...
-Uploading my_test_code_2-0.0.1-py3-none-any.whl...
+Uploading my_test_code-0.0.1-py3-none-any.whl...Uploading my_test_code_2-0.0.1-py3-none-any.whl...
Error: Command terminated with non-zero exit code 1 in benchmark iteration 54. Use the '-i'/'--ignore-failure' option if you want to ignore this. Alternatively, use the '--show-output' option to debug what went wrong.
```
An alternative could be to prepare a message fully in a local buffer and
write it in one call (I’m assuming underlying writer is still
synchronized). However, that’s more complicated and unclear if it’s
worth it, perf-wise.
## Tests
With this change I’m running the same hyperfine command with 1000
iterations with no failures.
## Changes
- Remove bundle.Parallel & bundle.ReadOnlyBundle.
- Add bundle.ApplyParallel, as a helper to migrate from bundle.Parallel.
- Keep ReadOnlyMutator as a separate type but it's now a subtype of
Mutator so it works on regular *Bundle. Having it as a separate type
prevents non-readonly mutators being passed to ApplyParallel
- validate.Validate becomes a function (was Mutator).
## Why
This a follow up to #2390 where we removed most of the tools to
construct chains of mutators. Same motivation applies here.
When it comes to read-only bundles, it's a leaky abstraction -- since
it's a shallow copy, it does not actually guarantee or enforce readonly
access to bundle. A better approach would be to run parallel operations
on independent narrowly-focused deep-copied structs, with just enough
information to carry out the task (this is not implemented here, but the
eventual goal). Now that we can just write regular code in phases and
not limited to mutator interface, we can switch to that approach.
## Tests
Existing tests.
---------
Co-authored-by: shreyas-goenka <88374338+shreyas-goenka@users.noreply.github.com>
## Changes
Rewrite bundle/tests/python_wheel_test.go into acceptance tests. The
same configs are used, but the test now runs 'bundle deploy' and in
addition to checking the files on the file system, also checks that the
files were uploaded and records jobs/create request.
There is a new test helper bin/find.py which filters out paths based on
regex, asserts on number of expected results. I've added it because
'find' on Windows behaves differently, so this helps avoid
cross-platform differences.
Bumps
[actions/upload-artifact](https://github.com/actions/upload-artifact)
from 4.6.0 to 4.6.1.
<details>
<summary>Release notes</summary>
<p><em>Sourced from <a
href="https://github.com/actions/upload-artifact/releases">actions/upload-artifact's
releases</a>.</em></p>
<blockquote>
<h2>v4.6.1</h2>
<h2>What's Changed</h2>
<ul>
<li>Update to use artifact 2.2.2 package by <a
href="https://github.com/yacaovsnc"><code>@yacaovsnc</code></a> in <a
href="https://redirect.github.com/actions/upload-artifact/pull/673">actions/upload-artifact#673</a></li>
</ul>
<p><strong>Full Changelog</strong>: <a
href="https://github.com/actions/upload-artifact/compare/v4...v4.6.1">https://github.com/actions/upload-artifact/compare/v4...v4.6.1</a></p>
</blockquote>
</details>
<details>
<summary>Commits</summary>
<ul>
<li><a
href="4cec3d8aa0"><code>4cec3d8</code></a>
Merge pull request <a
href="https://redirect.github.com/actions/upload-artifact/issues/673">#673</a>
from actions/yacaovsnc/artifact_2.2.2</li>
<li><a
href="e9fad966cc"><code>e9fad96</code></a>
license cache update for artifact</li>
<li><a
href="b26fd06e9d"><code>b26fd06</code></a>
Update to use artifact 2.2.2 package</li>
<li>See full diff in <a
href="65c4c4a1dd...4cec3d8aa0">compare
view</a></li>
</ul>
</details>
<br />
[](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores)
Dependabot will resolve any conflicts with this PR as long as you don't
alter it yourself. You can also trigger a rebase manually by commenting
`@dependabot rebase`.
[//]: # (dependabot-automerge-start)
[//]: # (dependabot-automerge-end)
---
<details>
<summary>Dependabot commands and options</summary>
<br />
You can trigger Dependabot actions by commenting on this PR:
- `@dependabot rebase` will rebase this PR
- `@dependabot recreate` will recreate this PR, overwriting any edits
that have been made to it
- `@dependabot merge` will merge this PR after your CI passes on it
- `@dependabot squash and merge` will squash and merge this PR after
your CI passes on it
- `@dependabot cancel merge` will cancel a previously requested merge
and block automerging
- `@dependabot reopen` will reopen this PR if it is closed
- `@dependabot close` will close this PR and stop Dependabot recreating
it. You can achieve the same result by closing it manually
- `@dependabot show <dependency name> ignore conditions` will show all
of the ignore conditions of the specified dependency
- `@dependabot ignore this major version` will close this PR and stop
Dependabot creating any more for this major version (unless you reopen
the PR or upgrade to it yourself)
- `@dependabot ignore this minor version` will close this PR and stop
Dependabot creating any more for this minor version (unless you reopen
the PR or upgrade to it yourself)
- `@dependabot ignore this dependency` will close this PR and stop
Dependabot creating any more for this dependency (unless you reopen the
PR or upgrade to it yourself)
</details>
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
## Changes
1. Add a Why section to the pull request template
2. Slightly improve language in the existing sections
## Why
Providing the right context for the reviewer in the PR description is
important as it usually cannot be inferred from the code itself. The new
section directly prompts the requester to provide such context
## Tests
Checked that the markdown is still rendered correctly in the local
viewer
Shreyas Goenka
shreyas-goenka
Denis Bilenko
Andrew Nester
Pieter Noordhuis
Russell Clarey
Anton Nekipelov
dependabot[bot]