## Changes
- Do not modify or edit .gitignore in bundle root.
- Instead create .databricks/.gitignore with content set to "*"
## Why
Merging our changes into existing .gitignore is complicated and adding
.gitignore where it's not expected adds to the noise. Other tools also
use the approach in this PR (e.g. ruff creates .ruff_cache/.gitignore).
## Tests
- Modified templates/default-sql to capture this new file.
## Changes
- Added `service-principal-secrets` command
- Added `budget-policy-id` for apps
- `experiments.log-inputs` now requires `ID` parameter as an input
- Added `genie.get-space` command
- Added `providers.list-provider-share-assets` command
For the whole list of SDK changes see:
https://github.com/databricks/databricks-sdk-go/releases/tag/v0.59.0
## Changes
Switch to use GET workspaces-files/{name} instead of workspace/export
for state files.
## Why
`/api/2.0./workspaces-files/{name}` has a higher limit which allows to
export state files larger than 10 MBs (which is the current limit for
`workspace/export`). We don't use the same API for read in other places
and fully replacing existing Filer because it doesn't correct get the
file content for notebooks and returns "File Not Found" error instead.
## Tests
All existing tests pass
## Changes
This function provides all environment variables necessary to
authenticate the downstream applications to the same credentials as the
parent process.
It's used in https://github.com/databricks/cli/pull/2278 and will also
be useful for the `databricks bundle exec` command.
## Tests
Unit test.
## Changes
This PR adds the auth.EnvVars function, which is a list of all
environment variables that the SDK uses to read auth configuration.
This is useful for spawning subprocesses since you can unset all auth
environment variables to clean up the environment before configuring the
auth.
It's used in #2278 today and will also be useful for the `databricks
bundle exec` command.
## Tests
Unit test.
## Changes
Add mutex synchronization in cmdio logger Log() method.
## Why
Since we issue multiple calls to underlying writer, we should lock the
whole method, otherwise we can get broken messages. One that can be
easily reproduced today is
```
hyperfine -m 100 --show-output 'go test ./acceptance -run ^TestAccept$/^bundle$/^artifacts$/^whl_multiple$ -count=1'
...
-Uploading my_test_code-0.0.1-py3-none-any.whl...
-Uploading my_test_code_2-0.0.1-py3-none-any.whl...
+Uploading my_test_code-0.0.1-py3-none-any.whl...Uploading my_test_code_2-0.0.1-py3-none-any.whl...
Error: Command terminated with non-zero exit code 1 in benchmark iteration 54. Use the '-i'/'--ignore-failure' option if you want to ignore this. Alternatively, use the '--show-output' option to debug what went wrong.
```
An alternative could be to prepare a message fully in a local buffer and
write it in one call (I’m assuming underlying writer is still
synchronized). However, that’s more complicated and unclear if it’s
worth it, perf-wise.
## Tests
With this change I’m running the same hyperfine command with 1000
iterations with no failures.
## Changes
This PR adds a warning which gives users clear guidance when they try to
use variable interpolation for an auth field.
## Tests
Modify existing acceptance test.
## Tests
Manually, I have a test that fails.
Before:
```
=== NAME TestAccept
server.go:195:
----------------------------------------
No stub found for pattern: GET /api/2.1/clusters/get
To stub a response for this request, you can add
the following to test.toml:
[[Server]]
Pattern = "GET /api/2.1/clusters/get"
Response.Body = '''
<response body here>
'''
Response.StatusCode = <response status-code here>
----------------------------------------
```
After:
```
server.go:203: No handler for URL: /api/2.1/clusters/get?cluster_id=0717-132531-5opeqon1
Body: [0 bytes]
For acceptance tests, add this to test.toml:
[[Server]]
Pattern = "GET /api/2.1/clusters/get"
Response.Body = '<response body here>'
# Response.StatusCode = <response code if not 200>
```
## Changes
- Instead of constructing chains of mutators and then executing them,
execute them directly.
- Remove functionality related to chain-building: Seq, If, Defer,
newPhase, logString.
- Phases become functions that apply the changes directly rather than
construct mutator chains that will be called later.
- Add a helper ApplySeq to call multiple mutators, use it where
Apply+Seq were used before.
This is intended to be a refactoring without functional changes, but
there are a few behaviour changes:
- Since defer() is used to call unlock instead of bundle.Defer()
unlocking will now happen even in case of panics.
- In --debug, the phase names are are still logged once before start of
the phase but each entry no longer has 'seq' or phase name in it.
- The message "Deployment complete!" was printed even if
terraform.Apply() mutator had an error. It no longer does that.
## Motivation
The use of the chains was necessary when mutators were returning a list
of other mutators instead of calling them directly. But that has since
been removed, so now the chain machinery have no purpose anymore.
Use of direct functions simplifies the logic and makes bugs more
apparent and easy to fix.
Other improvements that this unlocks:
- Simpler stacktraces/debugging (breakpoints).
- Use of functions with narrowly scoped API: instead of mutators that
receive full bundle config, we can use focused functions that only deal
with sections they care about prepareGitSettings(currentGitSection) ->
updatedGitSection. This makes the data flow more apparent.
- Parallel computations across mutators (within phase): launch
goroutines fetching data from APIs at the beggining, process them once
they are ready.
## Tests
Existing tests.
## Changes
Since at this moment we set default to 'no', interactively it should
also default to 'no'. However, it just uses the first option.
## Tests
Manually running `cli bundle init default-python`
## Changes
- Add 'serverless' prompt to default-python template (default is
currently set to "no").
- This is a simplified version of
https://github.com/databricks/cli/pull/2348 with 'auto' functionality
removed.
## Tests
- Split default-python into default-python/classic,
default-python/serverless, default-python/serverless-customcatalog.
- Manually check that "bundle init default-python" with serverless=yes
can be deployed and run on dogfood and test env.
## Changes
1. Change the **default-python** bundle template to set
`data_security_mode` of a cluster to SINGLE_USER
2. Change the **experimental-jobs-as-code** bundle template to set
`data_security_mode` of a cluster to SINGLE_USER
## Why
Explicitly adding this field saves experienced users from confusion onto
what security mode is applied to the cluster
## Tests
Changed existing unit and integration tests to pass with this change
## Changes
This PR:
1. No longer sets the `DATABRICKS_CLI_PARENT_PID` environment variable
since it was never required in the first place and was mistakenly merged
in the inital PR.
2. Performs minor cleanup based on post merge feedback in
https://github.com/databricks/cli/pull/2354.
## Tests
N/A
## Changes
Added missing .gitignore files to templates
## Tests
There were some incorrect snapshots of gitignore files in acceptance
tests, probably generated by testing infra. Updated them to new files
---------
Co-authored-by: Lennart Kats (databricks) <lennart.kats@databricks.com>
## Changes
- Instead of collecting requests in memory and writing them at the end
of the test, write them right away. Then test authors can do filtering
with jq in 'script' or collect individual files per different command.
- testserver is now simpler - it just calls a caller-provided function.
The logging logic is moved to acceptance_test.go.
See https://github.com/databricks/cli/pull/2359/files#r1967591173
## Tests
Existing tests.
## Changes
This removes the `run-as` property from the default templates. It's a
useful property but it still only works for jobs and it makes the
default databricks.yml a bit longer. It seems like users can just learn
about it from the docs and/or vary their deployment identity.
Depends on https://github.com/databricks/cli/pull/1712.
## Changes
This PR adds a library for spawning a daemon process. Our needs are
different from those of a typical daemon process in that we want to
handle being orphaned gracefully. This is because, in the vast majority
of telemetry use cases, the main CLI process (i.e., the parent process)
will exit before the telemetry process has a chance to finish uploading
the logs.
To achieve this we "detach" the child process from the parent process,
which requires different flags for UNIX vs. non-unix systems.
Here are the properties that we want to ensure for our telemetry
child/daemon processes:
1. They do not block the parent process.
Reason: The main CLI process should not be blocked on the telemetry
child process.
2. The child can read from stdin. The parent can write to stdin.
Reason: Telemetry logs will be passed to the child process via stdin.
3. Output logs do not leak from the child process.
Reason: Telemetry logs should not be visible to users of the CLI.
## Tests
Unit test
Shreyas Goenka
Russell Clarey
Denis Bilenko
Andrew Nester
shreyas-goenka
Anton Nekipelov
dependabot[bot]
Ilya Kuznetsov
Lennart Kats (databricks)