Commit Graph

1769 Commits

Author SHA1 Message Date
Anton Nekipelov 4f6649ffd7 (ci/cd debug) prepend experiment name with an extra slash to avoid replacements in windows environments 2025-03-11 17:25:57 +01:00
Anton Nekipelov 35d4a70739 (ci/cd debug) wrap bash variable into quotes to avoid replacements in windows environments 2025-03-11 17:04:04 +01:00
Anton Nekipelov ee69dbe00e (cleanup) remove unused bundle template 2025-03-11 15:54:25 +01:00
Anton Nekipelov 333a7cf1ea convert TestBindExperimentToExistingExperiment to an acceptance (cloud-only) test 2025-03-11 15:43:15 +01:00
Anton Nekipelov e563a0b0a3 fix comment line 2025-03-11 15:42:16 +01:00
Anton Nekipelov 7a98a7e4cd Add support for experiments in deployment bind/unbind commands 2025-03-11 15:42:16 +01:00
Denis Bilenko b30184f8c1
acc: add -tail option to see output.txt in real time (#2467)
## Changes
Add new option -tail to acceptance test runner. This logs output.txt in
real time with relative timestamp prefix.
 
## Why
When working on long running integration tests, it's useful to see what
stage they are at.

## Tests
Manually

```
~/work/cli/acceptance/selftest % testme -v -tail
+ go test .. -run ^TestAccept$/^selftest$ -v -tail
...
=== CONT  TestAccept/selftest/diff
    acceptance_test.go:774:  0.051 >>> diff.py out_dir_a out_dir_b
=== NAME  TestAccept/selftest/basic
    acceptance_test.go:774:  0.051 === Capturing STDERR
    acceptance_test.go:774:  0.051 >>> python3 -c import sys; sys.stderr.write("STDERR\n")
=== NAME  TestAccept/selftest/server
    acceptance_test.go:774:  0.050 >>> curl -s http://127.0.0.1:55850/api/2.0/preview/scim/v2/Me
...
    acceptance_test.go:774:  0.204 >>> /Users/denis.bilenko/work/cli/acceptance/build/darwin_arm64/databricks --version
=== NAME  TestAccept/selftest/server
    acceptance_test.go:774:  0.510 Error: Workspace path not found
...
```
2025-03-11 13:36:58 +00:00
dependabot[bot] e6ddb84964
build(deps): bump golang.org/x/text from 0.22.0 to 0.23.0 (#2456)
Bumps [golang.org/x/text](https://github.com/golang/text) from 0.22.0 to
0.23.0.
<details>
<summary>Commits</summary>
<ul>
<li><a
href="566b44fc93"><code>566b44f</code></a>
go.mod: update golang.org/x dependencies</li>
<li><a
href="d5156da34e"><code>d5156da</code></a>
collate/build: do not use println in tests</li>
<li><a
href="221d88cef7"><code>221d88c</code></a>
x/text: fix scientific notation by removing extraneous spaces</li>
<li><a
href="b18c10747b"><code>b18c107</code></a>
internal/export/unicode: change C comment to mention unassigned code
points</li>
<li><a
href="835f8ac434"><code>835f8ac</code></a>
language: use a more straightforward return value</li>
<li><a
href="ae68efb59e"><code>ae68efb</code></a>
internal/export/unicode: add CategoryAliases, Cn, and LC</li>
<li><a
href="518d9c0174"><code>518d9c0</code></a>
all: upgrade go directive to at least 1.23.0 [generated]</li>
<li>See full diff in <a
href="https://github.com/golang/text/compare/v0.22.0...v0.23.0">compare
view</a></li>
</ul>
</details>
<br />


[![Dependabot compatibility
score](https://dependabot-badges.githubapp.com/badges/compatibility_score?dependency-name=golang.org/x/text&package-manager=go_modules&previous-version=0.22.0&new-version=0.23.0)](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores)

Dependabot will resolve any conflicts with this PR as long as you don't
alter it yourself. You can also trigger a rebase manually by commenting
`@dependabot rebase`.

[//]: # (dependabot-automerge-start)
[//]: # (dependabot-automerge-end)

---

<details>
<summary>Dependabot commands and options</summary>
<br />

You can trigger Dependabot actions by commenting on this PR:
- `@dependabot rebase` will rebase this PR
- `@dependabot recreate` will recreate this PR, overwriting any edits
that have been made to it
- `@dependabot merge` will merge this PR after your CI passes on it
- `@dependabot squash and merge` will squash and merge this PR after
your CI passes on it
- `@dependabot cancel merge` will cancel a previously requested merge
and block automerging
- `@dependabot reopen` will reopen this PR if it is closed
- `@dependabot close` will close this PR and stop Dependabot recreating
it. You can achieve the same result by closing it manually
- `@dependabot show <dependency name> ignore conditions` will show all
of the ignore conditions of the specified dependency
- `@dependabot ignore this major version` will close this PR and stop
Dependabot creating any more for this major version (unless you reopen
the PR or upgrade to it yourself)
- `@dependabot ignore this minor version` will close this PR and stop
Dependabot creating any more for this minor version (unless you reopen
the PR or upgrade to it yourself)
- `@dependabot ignore this dependency` will close this PR and stop
Dependabot creating any more for this dependency (unless you reopen the
PR or upgrade to it yourself)


</details>

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2025-03-11 13:38:14 +01:00
Anton Nekipelov e2a5953568
Make bind/schema acceptance test run locally (#2463)
## Changes
<!-- Brief summary of your changes that is easy to understand -->
1. Change the cloud acceptance test for `bind/schema` to run locally
2. Add debug lines to the mock server
3. Change `fake_workspace` to create directories for imported files

## Why
<!-- Why are these changes needed? Provide the context that the reviewer
might be missing.
For example, were there any decisions behind the change that are not
reflected in the code itself? -->
1. Local version of the test run can indicate breaking changes faster
than the cloud version and it can be run locally without any predefined
environment variables

## Tests
<!-- How have you tested the changes? -->
1. Ran both acloud and local versions of test, both succeeded

<!-- If your PR needs to be included in the release notes for next
release,
add a separate entry in NEXT_CHANGELOG.md as part of your PR. -->
2025-03-11 12:25:32 +00:00
dependabot[bot] 6f9ddf4b20
build(deps): bump golang.org/x/oauth2 from 0.26.0 to 0.28.0 (#2459)
Bumps [golang.org/x/oauth2](https://github.com/golang/oauth2) from
0.26.0 to 0.28.0.
<details>
<summary>Commits</summary>
<ul>
<li><a
href="0042180b24"><code>0042180</code></a>
oauth2: Deep copy context client in NewClient</li>
<li><a
href="ce350bff61"><code>ce350bf</code></a>
oauth2: remove unneeded TokenSource implementation in transport
test</li>
<li><a
href="44967abe90"><code>44967ab</code></a>
google: fix typos</li>
<li><a
href="9c82a8cf7a"><code>9c82a8c</code></a>
oauth2.go: use a more straightforward return value</li>
<li><a
href="681b4d8edc"><code>681b4d8</code></a>
jws: split token into fixed number of parts</li>
<li><a
href="3f78298bee"><code>3f78298</code></a>
all: upgrade go directive to at least 1.23.0 [generated]</li>
<li><a
href="109dabf901"><code>109dabf</code></a>
endpoints: add links/provider for Discord</li>
<li><a
href="ac571fa341"><code>ac571fa</code></a>
oauth2: fix docs for Config.DeviceAuth</li>
<li><a
href="314ee5b92b"><code>314ee5b</code></a>
endpoints: add patreon endpoint</li>
<li>See full diff in <a
href="https://github.com/golang/oauth2/compare/v0.26.0...v0.28.0">compare
view</a></li>
</ul>
</details>
<br />


[![Dependabot compatibility
score](https://dependabot-badges.githubapp.com/badges/compatibility_score?dependency-name=golang.org/x/oauth2&package-manager=go_modules&previous-version=0.26.0&new-version=0.28.0)](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores)

Dependabot will resolve any conflicts with this PR as long as you don't
alter it yourself. You can also trigger a rebase manually by commenting
`@dependabot rebase`.

[//]: # (dependabot-automerge-start)
[//]: # (dependabot-automerge-end)

---

<details>
<summary>Dependabot commands and options</summary>
<br />

You can trigger Dependabot actions by commenting on this PR:
- `@dependabot rebase` will rebase this PR
- `@dependabot recreate` will recreate this PR, overwriting any edits
that have been made to it
- `@dependabot merge` will merge this PR after your CI passes on it
- `@dependabot squash and merge` will squash and merge this PR after
your CI passes on it
- `@dependabot cancel merge` will cancel a previously requested merge
and block automerging
- `@dependabot reopen` will reopen this PR if it is closed
- `@dependabot close` will close this PR and stop Dependabot recreating
it. You can achieve the same result by closing it manually
- `@dependabot show <dependency name> ignore conditions` will show all
of the ignore conditions of the specified dependency
- `@dependabot ignore this major version` will close this PR and stop
Dependabot creating any more for this major version (unless you reopen
the PR or upgrade to it yourself)
- `@dependabot ignore this minor version` will close this PR and stop
Dependabot creating any more for this minor version (unless you reopen
the PR or upgrade to it yourself)
- `@dependabot ignore this dependency` will close this PR and stop
Dependabot creating any more for this dependency (unless you reopen the
PR or upgrade to it yourself)


</details>

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2025-03-11 13:24:31 +01:00
dependabot[bot] 2fb67dc666
build(deps): bump golang.org/x/term from 0.29.0 to 0.30.0 (#2455)
Bumps [golang.org/x/term](https://github.com/golang/term) from 0.29.0 to
0.30.0.
<details>
<summary>Commits</summary>
<ul>
<li><a
href="04218fdaf7"><code>04218fd</code></a>
go.mod: update golang.org/x dependencies</li>
<li><a
href="208db03875"><code>208db03</code></a>
all: upgrade go directive to at least 1.23.0 [generated]</li>
<li>See full diff in <a
href="https://github.com/golang/term/compare/v0.29.0...v0.30.0">compare
view</a></li>
</ul>
</details>
<br />


[![Dependabot compatibility
score](https://dependabot-badges.githubapp.com/badges/compatibility_score?dependency-name=golang.org/x/term&package-manager=go_modules&previous-version=0.29.0&new-version=0.30.0)](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores)

Dependabot will resolve any conflicts with this PR as long as you don't
alter it yourself. You can also trigger a rebase manually by commenting
`@dependabot rebase`.

[//]: # (dependabot-automerge-start)
[//]: # (dependabot-automerge-end)

---

<details>
<summary>Dependabot commands and options</summary>
<br />

You can trigger Dependabot actions by commenting on this PR:
- `@dependabot rebase` will rebase this PR
- `@dependabot recreate` will recreate this PR, overwriting any edits
that have been made to it
- `@dependabot merge` will merge this PR after your CI passes on it
- `@dependabot squash and merge` will squash and merge this PR after
your CI passes on it
- `@dependabot cancel merge` will cancel a previously requested merge
and block automerging
- `@dependabot reopen` will reopen this PR if it is closed
- `@dependabot close` will close this PR and stop Dependabot recreating
it. You can achieve the same result by closing it manually
- `@dependabot show <dependency name> ignore conditions` will show all
of the ignore conditions of the specified dependency
- `@dependabot ignore this major version` will close this PR and stop
Dependabot creating any more for this major version (unless you reopen
the PR or upgrade to it yourself)
- `@dependabot ignore this minor version` will close this PR and stop
Dependabot creating any more for this minor version (unless you reopen
the PR or upgrade to it yourself)
- `@dependabot ignore this dependency` will close this PR and stop
Dependabot creating any more for this dependency (unless you reopen the
PR or upgrade to it yourself)


</details>

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2025-03-11 12:45:10 +01:00
Andrew Nester 676140c580
Set Go patch version in go.mod (#2466)
## Changes
Set Go patch version in go.mod

## Why

Our dependabot PRs failing with the following diff

```
Run # Exit with status code 1 if there are differences (i.e. unformatted files)
diff --git a/go.mod b/go.mod
index 5c1cb1d..493152a 100644
--- a/go.mod
+++ b/go.mod
@@ -1,6 +1,[7](https://github.com/databricks/cli/actions/runs/13768953030/job/38502532441?pr=2459#step:5:8) @@
 module github.com/databricks/cli
 
-go 1.23
+go 1.23.0
+
```

While I'm not sure why it started to happen, setting the patch version
at least should fix these PR.

This might be related to this change in Go
https://github.com/golang/go/issues/69095
2025-03-11 10:49:33 +00:00
Gleb Kanterov fdc6d12570
Update uv to 0.6.5 (#2449)
## Changes
Update uv to 0.6.5

## Tests
Using CI checks

---
NO_CHANGELOG=true
2025-03-11 10:30:03 +00:00
Gleb Kanterov 36431ef161
Setup CI for Python code (#2448)
## Changes
Add new CI jobs for testing Python code.

These jobs are separate from Golang code because they are executed
against a matrix of Python versions, and OS versions are not relevant.
We will get coverage with OS versions using acceptance tests. In
addition, we don't need to run these tests unless any of the Python code
is modified.

We use uv as a package manager to provision all CI tools:
- pyright to typecheck
- ruff to lint and format

experimental/python contains a Makefile with target similar to
repository root:
- make fmt
- make lint
- make build

## Tests
Using CI
<!-- How have you tested the changes? -->

---
NO_CHANGELOG=true
2025-03-11 10:29:36 +00:00
Anton Nekipelov eae60a797a
Make fake_workspace create all parent folders exist when importing a file (#2465)
## Changes
<!-- Brief summary of your changes that is easy to understand -->
When a file is imported, FakeWorkspace now automatically identifies and
records all parent directories in the directories map, ensuring the
directory structure is properly maintained without requiring explicit
directory creation.

## Why
<!-- Why are these changes needed? Provide the context that the reviewer
might be missing.
For example, were there any decisions behind the change that are not
reflected in the code itself? -->
During an acceptance test, production code might want to check a root
folder for existence, this change makes sure that root folder is marked
as existing when at least one file is imported into the workspace

## Tests
<!-- How have you tested the changes? -->
Tests for another change using this improvement are green:
https://github.com/databricks/cli/pull/2463

<!-- If your PR needs to be included in the release notes for next
release,
add a separate entry in NEXT_CHANGELOG.md as part of your PR. -->
2025-03-11 09:13:31 +00:00
Andrew Nester 8d7df68ac1
Update Go to 1.23.7 (#2461)
## Changes
Update Go to 1.23.7

From https://go.dev/doc/devel/release#go1.23.minor

> go1.23.5 (released 2025-01-16) includes security fixes to the
crypto/x509 and net/http packages, as well as bug fixes to the compiler,
the runtime, and the net package. See the [Go 1.23.5
milestone](https://github.com/golang/go/issues?q=milestone%3AGo1.23.5+label%3ACherryPickApproved)
on our issue tracker for details.

> go1.23.6 (released 2025-02-04) includes security fixes to the
crypto/elliptic package, as well as bug fixes to the compiler and the go
command. See the [Go 1.23.6
milestone](https://github.com/golang/go/issues?q=milestone%3AGo1.23.6+label%3ACherryPickApproved)
on our issue tracker for details.

> go1.23.7 (released 2025-03-04) includes security fixes to the net/http
package, as well as bug fixes to cgo, the compiler, and the reflect,
runtime, and syscall packages. See the [Go 1.23.7
milestone](https://github.com/golang/go/issues?q=milestone%3AGo1.23.7+label%3ACherryPickApproved)
on our issue tracker for details.
2025-03-10 15:40:11 +00:00
shreyas-goenka c19c8bbbf7
Add library to store cmd-exec-id in context (#2439)
## Why
The command execution ID is a UUID that should be initialized once, and
then have a consistent read value for the duration of the command.

As we start emitting telemetry events, we also want to emit the command
execution ID in the telemetry payload. This library helps consolidate
all reads to the `cmd-exec-id` value, and ensures there's only one write
for the UUID.

relevant comment:
https://github.com/databricks/cli/pull/2432#discussion_r1982884235

## Tests
Unit tests.
2025-03-10 13:11:24 +00:00
Hari Selvarajan fa0a734b3c
Enable offline install of labs projects (#2049)
## Changes
<!-- Summary of your changes that are easy to understand -->
This PR makes changes to the labs code base to allow for offline
installation of labs projects (like UCX). By passing a flag
--offline=true, the code will skip checking for project versions and
download code from GitHub and instead will look from the local
installation folder. This cmd is useful in systems where there is
internet restriction, the user should follow a set-up as follows:

- install a labs project on a machine which has internet
- zip and copy the file to the intended machine and 
- run databricks labs install <project name>--offline=true
it will look for the code in the same install directory and if present
load from there.


Closes #1646 

related to https://github.com/databrickslabs/ucx/issues/3418
## Tests
<!-- How is this tested? -->

Added unit test case and tested.

NO_CHANGELOG=true

---------

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: Pieter Noordhuis <pieter.noordhuis@databricks.com>
Co-authored-by: Lennart Kats (databricks) <lennart.kats@databricks.com>
Co-authored-by: Denis Bilenko <denis.bilenko@databricks.com>
Co-authored-by: Julia Crawford (Databricks) <julia.crawford@databricks.com>
Co-authored-by: Ilya Kuznetsov <ilya.kuznetsov@databricks.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Andrew Nester <andrew.nester@databricks.com>
Co-authored-by: Anton Nekipelov <226657+anton-107@users.noreply.github.com>
Co-authored-by: shreyas-goenka <88374338+shreyas-goenka@users.noreply.github.com>
2025-03-10 10:01:17 +00:00
Andrew Nester b5a7023ef1
Removed Changelog check Github Action (#2453)
## Changes
Removed Changelog check Github Action

## Why
Check for the Changelog and the necessity to add `NO_CHANGELOG=true`
appears to be quite noisy, so removing this

NO_CHANGELOG=true
2025-03-10 09:12:46 +00:00
Andrew Nester 77f6a285be
Fixed "can't evaluate field Name in type interface{}" for `databricks queries list` (#2451)
## Changes
Fixes #1888

## Why
`databricks queries list` is using override to define a template to
display a list in a nice table form. This Go template was referencing to
non existing anymore fields and hence failing.

## Tests
Added acceptance test
2025-03-10 08:05:15 +00:00
Ilya Kuznetsov 0c809db82a
Adapt docsgen output for new documentation framework (#2262)
## Changes

Update `docsgen` output to align with new documentation framework

1. New header style, previously we had brackets for `<name>`, now it's
in italic `_name_`
2. Updated broken markdown in OpenAPI descriptions
3. Table markdown has different structure in new framework

## Tests

Updated existing tests

NO_CHANGELOG=true
2025-03-07 18:11:08 +00:00
shreyas-goenka af3914db61
Move `WorkspaceClient` to `libs/command` (#2444)
## Changes
Move the WorkspaceClient reader and setter from the root package to
`libs/command`.

## Why
This allows us to read the workspace client set in the context in all
CLI packages. This was not possible before because using
`root.WorkspaceClient` would often result in an import cycle.

This would also allow us to standardise reads of the workspace client to
be from the context in bundle commands. Today those are read from the
bundle object tree instead.

This is a natural followup to
https://github.com/databricks/cli/pull/2440.

## Tests
Existing tests and one new unit test.

NO_CHANGELOG=true
2025-03-07 16:47:19 +00:00
Anton Nekipelov 0225c77b68
Add acceptance test for binding/unbinding schemas (#2441)
## Changes
<!-- Brief summary of your changes that is easy to understand -->
1. Created a new acceptance test for binding an existing schema to a
bundle
2. Added a feature flag support for acceptance test runner

## Why
<!-- Why are these changes needed? Provide the context that the reviewer
might be missing.
For example, were there any decisions behind the change that are not
reflected in the code itself? -->
1. The test is more simple than an existing integration test for bind
2. The test only can run in environments where Unity Catalog is enabled

## Tests
<!-- How have you tested the changes? -->
New test is passing in CI/CD

## Changelog
NO_CHANGELOG=true
2025-03-07 16:41:25 +00:00
shreyas-goenka 5948339f92
Add unit test for `command.ConfigUsed` (#2446)
## Changes
I forgot to add a unit test in the original PR, so adding it here:
https://github.com/databricks/cli/pull/2440

## Why
Unit testing code is a good practice. I believe we all should unit test
code.

## Tests
N/A

NO_CHANGELOG=true
2025-03-07 11:42:43 +00:00
Andrew Nester 1a02422d56
Make NO_CHANGELOG=true a default in PR template (#2445)
## Changes
Make NO_CHANGELOG=true a default in PR template

## Why
Quite large amount of our changes are internal and don't need call outs
in CHANGELOG hence this is a good default for majority of our PRs.
Without NO_CHANGELOG=true CI check expects to have an entry in
NEXT_CHANGELOG file as part of same PR.

NO_CHANGELOG=true
2025-03-06 18:03:17 +00:00
Denis Bilenko 0356e30965
acc: Confirm that included variable def overrides databricks.yml (#2442)
Expand the test a little to check that complex variables are fully
replaced (not merged) by definitions in include file.

NO_CHANGELOG=true
2025-03-06 17:21:59 +00:00
shreyas-goenka 10b245d10c
Move `ConfigUsed` to `libs/command` (#2440)
## Changes
This PR moves `ConfigUsed` from the root package to `libs/command`.

## Why
Having the ConfigUsed function in the root package is a problem because
that means we cannot use that function from outside the `root` package
since doing so often leads to an import cycle (because `root` imports
everything implicitly).

Moving it to a separate package that consolidates the read interaction
and solves the import cycle issue. Example where this would have
simplified code:
https://github.com/databricks/cli/pull/2432#discussion_r1983368092

I'd like to send PRs to do the same for the workspace client and account
client as well. I'll wait however until this one is merged incase people
have concerns about the approach here.

## Tests
Existing tests.
2025-03-06 17:08:55 +00:00
hectorcast-db c6e2be1ecb
Enable Automated tagging workflow (#2361)
## Changes
Enable Automated tagging workflow. The workflow uses the `tagging.py`
script to:
* Flush the NEXT_CHANGELOG.md` file into the `CHANGELOG.md`.
* Create a Release commit and push it to `main` (without a PR)
* Create a tag for the release.

Usage and Changelog management details can be found in
[go/deco/release](go/deco/release).

This PR does:
* Added `cli_legacy` mode. Each SDK has it's own "mode" used to
determine which templates are to be used.
* `api_changelog: true` enables API changelog to be appended to
NEXT_CHANGELOG when generating the SDK.
* Removed `.codegen/changelog.md.tmpl`. Changelog templates are now part
of universe.
* Workflow and `tagging.py` files are synced from universe as part of
SDK generation. This allows us to test them and keep them in sync with
other SDKs.

## Tests
Workflow enabled in databricks/terraform-provider-databricks

---------

Co-authored-by: Andrew Nester <andrew.nester@databricks.com>
2025-03-06 15:30:01 +00:00
shreyas-goenka 897741f55a
Store DBR version in context (#2438)
## Changes
This PR also starts storing the DBR versions in the context.

Go patch file used:
```
@@
var x expression
@@
-dbr.MockRuntime(x, true)
+dbr.MockRuntime(x, dbr.Environment{IsDbr: true, Version: "15.4"})

@@
var x expression
@@
-dbr.MockRuntime(x, false)
+dbr.MockRuntime(x, dbr.Environment{})
```

ref: https://github.com/uber-go/gopatch

## Why
This localised all DBR version accesses to `libs/dbr`. Relevant comment:
https://github.com/databricks/cli/pull/2432#discussion_r1982878616

## Tests
Exiting tests are modified.
2025-03-06 13:01:43 +00:00
Denis Bilenko edf37e7d0d
Add libs/patchwheel: insert dynamic version in whl (#2427)
## Changes
- New library to patch existing whl with dynamic version. The version is
based on mtime of original wheel.
- New hidden command "selftest patchwhl" (for acceptance test).

## Why
Part of the project to deprecate dynamic version in setup.py template
(#1034). This would enable switching default-python to pyproject.toml
without dependency on setuptools-specific sections.

The dynamic version is needed to ensure that wheels are picked up by the
users code when doing development.

## Tests
- New unit tests that use uv to create envs on various versions and
install (and re-install) patched wheels there.
- New acceptance test that patches prebuilt whl and records the result
fully unzipped.
- New acceptance test helper setmtime.py to set mtime with nanosecond
precision cross-platform.
2025-03-06 10:28:30 +00:00
Andrew Nester c38efaa5c4
[Release] Release v0.243.0 (#2435)
CLI:
* Upgrade Go SDK to 0.59.0
([#2425](https://github.com/databricks/cli/pull/2425)).

Bundles:
* Added a warning when `config` section is used in apps
([#2416](https://github.com/databricks/cli/pull/2416)).
* Switch to use GET workspaces-files/{name} instead of workspace/export
for state files to avoid 10MB limit
([#2423](https://github.com/databricks/cli/pull/2423)).
* Use schema field for pipeline in builtin template
([#2347](https://github.com/databricks/cli/pull/2347)).
* Add warning when variable interpolation is used for auth fields
([#2399](https://github.com/databricks/cli/pull/2399)).
* Add warning when include is used in config files other than
databricks.yml ([#2389](https://github.com/databricks/cli/pull/2389)).
* Add support for schemas in deployment bind/unbind commands
([#2406](https://github.com/databricks/cli/pull/2406)).
* Do not modify/create .gitignore in bundle root
([#2429](https://github.com/databricks/cli/pull/2429)).
* Raise an error when there are multiple local libraries with the same
basename used ([#2382](https://github.com/databricks/cli/pull/2382)).
* Upgrade TF provider to 1.68.0
([#2426](https://github.com/databricks/cli/pull/2426)).

API Changes:
* Changed `databricks experiments log-inputs` command with new required
argument order.
 * Added `databricks genie get-space` command.
 * Added `databricks providers list-provider-share-assets` command.
* Changed `databricks shares update-permissions` command return type to
become non-empty.

OpenAPI commit e5c870006a536121442cfd2441bdc8a5fb76ae1e (2025-03-03)
2025-03-05 15:42:40 +00:00
shreyas-goenka a24a7f5738
Remove omitempty tag for exit code and execution time in telemetry (#2433) 2025-03-05 14:20:59 +00:00
Pieter Noordhuis 6ae353d84b
Use schema field for pipeline in builtin template (#2347)
## Changes

The `schema` field implies the lifecycle of tables is no longer tied to
the lifecycle of the pipeline, as was the case with the `target` field.

More information about using the "catalog" and "schema" properties can
be found here:
https://docs.databricks.com/en/delta-live-tables/target-schema.html

## Tests

n/a

---------

Co-authored-by: Lennart Kats (databricks) <lennart.kats@databricks.com>
2025-03-05 14:19:33 +00:00
Russell Clarey 3a3076d9ea
Output deprecated annotations in the JSON schema (#2422)
## Changes
Start outputting `deprecated` and `deprecationMessage` annotations in
the JSON schema

## Why
So that deprecated fields are shown as deprecated in VSCode (and other
editors)

## Tests
- manually tested

||||
|---|---|---|
|<img width="682" alt="Screenshot 2025-03-03 at 16 04 21"
src="https://github.com/user-attachments/assets/9712aa2f-0f41-48a9-8bd0-ec92b8b75c3f"
/>|<img width="682" alt="Screenshot 2025-03-03 at 16 04 29"
src="https://github.com/user-attachments/assets/a14f3523-ad85-4fed-96d7-2a8cf3a458e5"
/>|<img width="682" alt="Screenshot 2025-03-03 at 16 06 03"
src="https://github.com/user-attachments/assets/b9ab1050-048e-4c14-b183-96d615a4fbc1"
/>|
2025-03-05 11:17:03 +00:00
Denis Bilenko a26461c5a3
Do not modify/create .gitignore in bundle root (#2429)
## Changes
- Do not modify or edit .gitignore in bundle root.
- Instead create .databricks/.gitignore with content set to "*"

## Why
Merging our changes into existing .gitignore is complicated and adding
.gitignore where it's not expected adds to the noise. Other tools also
use the approach in this PR (e.g. ruff creates .ruff_cache/.gitignore).

## Tests
- Modified templates/default-sql to capture this new file.
2025-03-05 10:57:05 +00:00
Andrew Nester 4dba35dff4
Upgrade TF provider to 1.68.0 (#2426)
## Changes
Upgrade TF provider to 1.68.0

- Added support for Volumes as a destination for Cluster log conf
2025-03-05 10:20:55 +00:00
Andrew Nester 294db2ecca
Upgrade Go SDK to 0.59.0 (#2425)
## Changes
- Added `service-principal-secrets` command
- Added `budget-policy-id` for apps
- `experiments.log-inputs` now requires `ID` parameter as an input
- Added `genie.get-space` command
- Added `providers.list-provider-share-assets` command

For the whole list of SDK changes see:
https://github.com/databricks/databricks-sdk-go/releases/tag/v0.59.0
2025-03-05 10:20:51 +00:00
shreyas-goenka b21fdac209
Update VSCode settings to resolve files named `script` as shell scripts (#2419)
## Why
VSCode for me resolves `script` as YAML files showing incorrect syntax
coloring. This setting forces VSCode to treat all files named `script`
as shell scripts leading to better syntax highlighting.

These files are common place in our acceptance tests.

## Tests
Manually.
2025-03-05 10:02:40 +00:00
Denis Bilenko 1fbb81eb87
Disable ruff cache (#2430)
There is no perf difference, it's fast in both cases:

```
~/work/cli % hyperfine 'ruff format -qn'
Benchmark 1: ruff format -qn
  Time (mean ± σ):      22.5 ms ±   4.1 ms    [User: 23.6 ms, System: 106.7 ms]
  Range (min … max):    17.3 ms …  50.2 ms    52 runs
```

but it no longer produces .ruff_cache which can confuse acceptance test
runner

```
~/work/cli % find . -type d | grep ruff
./.ruff_cache
./.ruff_cache/0.8.3
./libs/notebook/testdata/.ruff_cache
./libs/notebook/testdata/.ruff_cache/0.8.3
./libs/sync/testdata/.ruff_cache
./libs/sync/testdata/.ruff_cache/0.8.3
./acceptance/bundle/templates/default-sql/.ruff_cache
./acceptance/bundle/templates/default-sql/.ruff_cache/0.8.3
```
2025-03-05 08:45:51 +01:00
Andrew Nester 41961226be
Switch to use GET workspaces-files/{name} instead of workspace/export for state files (#2423)
## Changes
Switch to use GET workspaces-files/{name} instead of workspace/export
for state files.

## Why
`/api/2.0./workspaces-files/{name}` has a higher limit which allows to
export state files larger than 10 MBs (which is the current limit for
`workspace/export`). We don't use the same API for read in other places
and fully replacing existing Filer because it doesn't correct get the
file content for notebooks and returns "File Not Found" error instead.

## Tests
All existing tests pass
2025-03-04 15:03:51 +00:00
shreyas-goenka 549b226cbc
Add the `auth.ProcessEnv` function (#2404)
## Changes
This function provides all environment variables necessary to
authenticate the downstream applications to the same credentials as the
parent process.

It's used in https://github.com/databricks/cli/pull/2278 and will also
be useful for the `databricks bundle exec` command.

## Tests
Unit test.
2025-03-04 14:47:11 +00:00
Anton Nekipelov c0f5436a28
Add support for schemas in deployment bind/unbind commands (#2406)
## Changes
1. Changed `FindResourceByConfigKey` to return schema resources
2. Implemented the `Exists` method for schema resources

## Why
This PR adds support for schema resources in deployment operations,
enabling users to:
- Bind schemas using `databricks bundle deployment bind <myschema_key>
<schema_full_name>`
- Unbind schemas using `databricks bundle deployment unbind
<myschema_key>`

Where:
- `myschema_key` is a resource key defined in the bundle's .yml file
- `schema_full_name` references an existing schema in the Databricks
workspace

These capabilities allow for more flexible resource management of
schemas within bundles.

## Tests

Added a new integration test that tests bind and unbind methods together
with bundle deployment and destruction.
2025-03-04 12:46:39 +00:00
Anton Nekipelov bcce6b0ece
Enable TestAuthDescribeFailure integration test (#2420)
## Changes
1. remove t.Skip directive from TestAuthDescribeFailure integration test
2. remove checking the host in the test

## Why
1. This enables integration test that exercises `databricks auth
describe` command, which was previously throwing errors in the CI/CD
pipeline. For the nonexistent profile check to fail, we need to create a
new profile file for the CLI to check (otherwise the configuration skips
checking of the provided profile actually exists).
2. I removed the assertion for the host part because it is not relevant
for the test, and setting that up just required more code in the test
setup.

## Tests
The integration test is passing
2025-03-04 11:43:26 +00:00
Andrew Nester 010f88f84e
Added a warning when `config` section is used in apps (#2416)
## Changes
Added a warning when `config` section is used in apps

## Why
To avoid the confusion between using apps in DABs and outside of DABs,
we want to provide only one way of configuring apps runtime
configuration - by using `app.yml` file in the root of the app.

## Tests
Added acceptance tests
2025-03-03 16:40:28 +00:00
Andrew Nester 8b7e0ba683
test: Lock setuptools version in TestDefaultPython test (#2421)
## Changes
Lock setuptools version to 75.8.2 (latest as of March 3, 2025)

## Why
As part of the tests `uv install` was installing latest version of
setuptools which led to all tests started to fail on Feb 25 when 75.8.1
setuptools version was released and which changed the naming of the
output built artifacts
https://setuptools.pypa.io/en/stable/history.html#v75-8-1

This change prevents us from breakages like this.

## Tests
Existing tests pass
2025-03-03 15:53:58 +00:00
Andrew Nester 3b07265113
Restrict same name libraries check for only whl and jar types (#2401)
## Changes
Same name libraries check only valid for local libraries. Local
libraries are only supported for Whl and Jar types. Hence we can
restrict matching pattern only to these libraries.

## Tests
Existing acceptance tests pass
2025-03-03 15:34:41 +00:00
shreyas-goenka 807a37b36a
Add the `auth.EnvVars` function (#2403)
## Changes
This PR adds the auth.EnvVars function, which is a list of all
environment variables that the SDK uses to read auth configuration.

This is useful for spawning subprocesses since you can unset all auth
environment variables to clean up the environment before configuring the
auth.

It's used in #2278 today and will also be useful for the `databricks
bundle exec` command.
 
## Tests
Unit test.
2025-03-03 14:58:43 +00:00
Denis Bilenko 5c146ca57a
Synchronize logging in cmdio (#2418)
## Changes
Add mutex synchronization in cmdio logger Log() method.

## Why

Since we issue multiple calls to underlying writer, we should lock the
whole method, otherwise we can get broken messages. One that can be
easily reproduced today is

```
 hyperfine -m 100 --show-output 'go test ./acceptance -run ^TestAccept$/^bundle$/^artifacts$/^whl_multiple$ -count=1'
...
            -Uploading my_test_code-0.0.1-py3-none-any.whl...
            -Uploading my_test_code_2-0.0.1-py3-none-any.whl...
            +Uploading my_test_code-0.0.1-py3-none-any.whl...Uploading my_test_code_2-0.0.1-py3-none-any.whl...

Error: Command terminated with non-zero exit code 1 in benchmark iteration 54. Use the '-i'/'--ignore-failure' option if you want to ignore this. Alternatively, use the '--show-output' option to debug what went wrong.
```

An alternative could be to prepare a message fully in a local buffer and
write it in one call (I’m assuming underlying writer is still
synchronized). However, that’s more complicated and unclear if it’s
worth it, perf-wise.

## Tests
With this change I’m running the same hyperfine command with 1000
iterations with no failures.
2025-03-03 13:42:43 +00:00
Denis Bilenko e4cd782852
Remove bundle.{Parallel,ReadOnlyBundle} (#2414)
## Changes
- Remove bundle.Parallel & bundle.ReadOnlyBundle.
- Add bundle.ApplyParallel, as a helper to migrate from bundle.Parallel.
- Keep ReadOnlyMutator as a separate type but it's now a subtype of
Mutator so it works on regular *Bundle. Having it as a separate type
prevents non-readonly mutators being passed to ApplyParallel
- validate.Validate becomes a function (was Mutator).

## Why
This a follow up to #2390 where we removed most of the tools to
construct chains of mutators. Same motivation applies here.

When it comes to read-only bundles, it's a leaky abstraction -- since
it's a shallow copy, it does not actually guarantee or enforce readonly
access to bundle. A better approach would be to run parallel operations
on independent narrowly-focused deep-copied structs, with just enough
information to carry out the task (this is not implemented here, but the
eventual goal). Now that we can just write regular code in phases and
not limited to mutator interface, we can switch to that approach.

## Tests
Existing tests.

---------

Co-authored-by: shreyas-goenka <88374338+shreyas-goenka@users.noreply.github.com>
2025-03-03 13:35:36 +00:00
Denis Bilenko 2c5b61538d
acc: Prevent regex in test.toml from consuming too much (#2417)
The original regex can consume both lines and sometimes it does.
2025-03-03 12:55:52 +00:00