## Changes
1. Change the **default-python** bundle template to set
`data_security_mode` of a cluster to SINGLE_USER
2. Change the **experimental-jobs-as-code** bundle template to set
`data_security_mode` of a cluster to SINGLE_USER
## Why
Explicitly adding this field saves experienced users from confusion onto
what security mode is applied to the cluster
## Tests
Changed existing unit and integration tests to pass with this change
## Changes
Added PyPi and Maven libraries tests
Needed for this PR since we don't currently have any coverage for PyPi
or Maven libraries
https://github.com/databricks/cli/pull/2382
## Changes
Previously using python wheel tasks in the tasks with compute referering
to interactive cluster defied in the same bundle would produce a warning
like below
```
GET /api/2.1/clusters/get?cluster_id=${resources.clusters.development_cluster.id}
< HTTP/2.0 400 Bad Request
< {
< "error_code": "INVALID_PARAMETER_VALUE",
< "message": "Cluster ${resources.clusters.development_cluster.id} does not exist"
< } pid=14465 mutator=seq mutator=initialize mutator=seq mutator=PythonWrapperWarning sdk=true
```
This PR fixes it by making sure that we check spark version for such
clusters based on its bundle configuration and don't make API calls
## Tests
Added acceptance test
## Changes
CLI generation template was using RequiredPathField from incorrect
request entity (body field from request and not request itself). Thus
for some of the commands required path parameters were not required when
--json was specified.
## Tests
Regenerated commands work correctly
## Changes
Added missing .gitignore files to templates
## Tests
There were some incorrect snapshots of gitignore files in acceptance
tests, probably generated by testing infra. Updated them to new files
---------
Co-authored-by: Lennart Kats (databricks) <lennart.kats@databricks.com>
## Changes
Previously, one could not set `LocalOnly=true` in parent directory and
then override it with `LocalOnly=false` in child directory. This is
because, `false` is considered empty value by mergo.
In order to distinguish between 'explicitly set to false' and 'not set',
I've changed all simple variables in config to be pointers. Now, one can
always override those, because non-nil pointers are not null (with
mergo.WithoutDereference option).
## Tests
Manually:
```
~/work/cli/acceptance/bundle/templates/default-python % cat test.toml # add this new file
LocalOnly = false
~/work/cli/acceptance/bundle/templates/default-python % CLOUD_ENV=aws go test ../../.. -run ^TestAccept$/^bundle$/^templates$/^default-python$ -v
(the test is no longer skipped)
```
## Changes
- Instead of collecting requests in memory and writing them at the end
of the test, write them right away. Then test authors can do filtering
with jq in 'script' or collect individual files per different command.
- testserver is now simpler - it just calls a caller-provided function.
The logging logic is moved to acceptance_test.go.
See https://github.com/databricks/cli/pull/2359/files#r1967591173
## Tests
Existing tests.
## Changes
This removes the `run-as` property from the default templates. It's a
useful property but it still only works for jobs and it makes the
default databricks.yml a bit longer. It seems like users can just learn
about it from the docs and/or vary their deployment identity.
Depends on https://github.com/databricks/cli/pull/1712.
## Changes
diff.py is like "diff -r -U2" but it applies replacements first to the
argument.
This allows comparing different output files and directories but ignore
differences that are going to be replaced by placeholders.
This is useful for tests that record large amount of files, specifically
"bundle init" with standard templates. In those tests, changing one
parameter results in a small diff so recording the full directory is not
helpful, because it's hard to see what changed there. I'm using it in
implementation of serverless mode for templates that need it: #2348 The
serverless templates are slightly different from classic, capturing the
diff helps to see exactly where.
Related small changes:
- Add [TESTROOT] replacement for absolute path to acceptance directory
in git repo.
- Add $TESTDIR env var for absolute path to a given test in git repo.
## Tests
- New test acceptance/selftest/diff to test the helper.
- Via #2348 which makes use of this feature.
This is needed for b.WorkspaceClient().CurrentWorkspaceID(ctx) which is
used by initialize_urls.go mutator ("bundle summary") #2316
It also also needed for to call serverless detection endpoint #2348
Builds on top of #2338
## Changes
These changes break the use of non-local libraries (such as PyPI
libraries).
This reverts the set so we can cut a patch release and take a closer
look later.
Original PRs are #2297 and #2341.
Issue reported in #2343.
## Tests
Manually confirmed that a bundle with PyPI package in libraries now
deploys fine.
## Changes
The CheckForSameNameLibraries mutator incorrectly assumed all resource
libraries define libraries as paths of the `string` type, but some
libraries, such as PyPi and Maven, define them as objects.
This PR addresses this issue. It was introduced in #2297.
## Tests
Added regression test.
## Changes
Handlers now receive testserver.Request and return any which could be
- string or []byte (returns it as is but sets content-type to json or
plain text depending on content)
- struct (encodes it as json and sets content-type to json)
- testserver.Response (full control over status and headers)
Note if testserver.Response is returned from the handler, it's Body
attribute can still be an object. In that case, it'll be serialized and
appropriate content-type header will be added.
The config is now using the same testserver.Response struct, the same
logic applies both configured responses and responses returned from
handlers.
As a result, one can set headers both in Golang handlers and in
test.toml.
This also fixes a bug with RecordRequest not seeing the body if it was
already consumed by the handler.
## Tests
- Existing rests.
- acceptance/selftest/server is extended to set response header.
## Changes
Reverse the order of stubs to match expectation (leaf configuration
takes precedence over parent configuration).
Follow up to #2334 .
## Tests
acceptance/selftest/server is extended with duplicate handler
## Changes
This PR adds acceptance tests for native Databricks auth methods: basic,
oauth, and pat.
In the future we could compare this with auth credentials used by
downstream tools like TF or the telemetry process to ensure consistent
auth credentials are picked up and used.
Note:
We do not add acceptance tests for other auth methods like Azure because
they communicate with external endpoints. To test them locally, we would
need to set up a reverse proxy server, which is out of scope for this
change.
## Tests
N/A
## Changes
- Currently if you define [[Server]] block, you disable the default
server implementation. With this change [[Server]] block takes
precedence over default server but default server remains.
- Switched mux implementation to
[gorilla/mux](https://github.com/gorilla/mux) -- unlike built-in it does
not panic if you set two handlers on the same part (instead the earliest
one wins). It also does not have any dependencies.
- Move acceptance/selftest into acceptance/selftest/basic and added
acceptance/selftest/server that demoes server override.
- Rewrite server set up to ensure that env vars and replacements are set
up correctly. Previously replacements for DATABRICKS_HOST referred to
default server, not to the custom server.
- Avoid calling CurrentUser.Me() in the local case. This allows
overriding /api/2.0/preview/scim/v2/Me, which we use in some tests (e.g.
bundle/templates-machinery/helpers-error). Previously the test passed
because CurrentUser.Me() was calling default server which is incorrect
but it happened to make the tests pass.
- The default server is now available on DATABRICKS_DEFAULT_HOST env
var.
- Rewrite "not found" handler in local test to handle error better (do
not raise http500 when header is already written).
## Tests
New acceptance test selftest/server specifically tests that both custom
and default handlers are available in a single test.
## Changes
- Only print replacements if VERBOSE_TEST flag is set.
- This is set on CI but not when you do "go test" or "make test".
Note, env var is used, so that it can be set in Makefile.
## Tests
Manually.
## Changes
Upgrade to TF provider 1.65.1
Notable changes:
- Now it's possible to use `run_as` field in `pipelines` definition
- Added support for `performance_target` for `jobs`
## Changes
- Insert newline after rendering indented JSON in bundle
validate/summary/run.
- This prevents "No newline at end of file" message in various cases,
for example when switching between recording raw output of the command
to output processed by jq, since jq does add a newline or when running
diff in acceptance tests.
## Tests
Manually running validate:
```
~/work/dabs_cuj_brickfood % ../cli/cli-main bundle validate -o json | tail -n 2 # without change
Error: root_path must start with '~/' or contain the current username to ensure uniqueness when using 'mode: development'
}
}%
~/work/dabs_cuj_brickfood % ../cli/cli bundle validate -o json | tail -n 2 # with change
Error: root_path must start with '~/' or contain the current username to ensure uniqueness when using 'mode: development'
}
}
~/work/dabs_cuj_brickfood %
```
Via #2316 -- see cleaner output there.
The tests in acceptance/bundle/template focus on standard templates. The
plan is to extend them with "bundle deploy" and enable them on the
cloud.
The tests in acceptance/bundle/template-machinery focus on specific
aspects of template implementation. Most of them are expected to remain
local-only.
## Changes
When there are many unexpected files, it's good to see them as a list
rather than scattered throughout the output.
<!-- Summary of your changes that are easy to understand -->
## Tests
Manually, example output:
```
acceptance_test.go:363: Test produced unexpected files:
output/my_default_sql/.databricks/bundle/dev/sync-snapshots/71c79ded90615dc7.json
output/my_default_sql/.databricks/bundle/dev/terraform/.terraform/providers/registry.terraform.io/databricks/databricks/1.64.1/darwin_arm64
output/my_default_sql/.databricks/bundle/dev/terraform/plan
output/my_default_sql/.databricks/bundle/prod/sync-snapshots/83e677e75259c93b.json
output/my_default_sql/.databricks/bundle/prod/terraform/.terraform/providers/registry.terraform.io/databricks/databricks/1.64.1/darwin_arm64
```
For acceptance/bundle/templates I'd like to run "bundle deploy". This
would create .databricks directory inside materialized output. It might
makes sense to commit some of this as part of golden files output. Even
if we did not commit anything, the test runner will see those files and
show the difference. Thus git should also see them.
Also rename .gitignore to out.gitignore in those tests, since that
includes .databricks as well.
## Changes
This will generate bundle YAML configuration for Git based jobs but
won't download any related files as they are in Git repo.
Fixes#1423
## Tests
Added unit test
---------
Co-authored-by: Pieter Noordhuis <pieter.noordhuis@databricks.com>
## Changes
Instead of using leaf-most config, all configs from root at
acceptance/test.toml to all intermediate ones to leaf config are merged
into one. Maps are merged, slices are appended, other values are
overridden.
I had to disable caching, because it is tricky when merging is involved
- deep copy is needed. There is performance
impact but currently it is tiny, about 1%.
Also, remove empty root config.
## Tests
Manually checked that inheritance of LocalOnly setting worked for these
tests:
Before - integration tests showed:
```
PASS acceptance.TestAccept/bundle/templates/wrong-url (0.70s)
PASS acceptance.TestAccept/bundle/templates/wrong-path (0.44s)
```
After:
```
SKIP acceptance.TestAccept/bundle/templates/wrong-url (0.00s)
SKIP acceptance.TestAccept/bundle/templates/wrong-path (0.00s)
acceptance_test.go:216: Disabled via LocalOnly setting in bundle/templates/test.toml, bundle/templates/wrong-path/test.toml (CLOUD_ENV=***)
```
## Changes
This PR fails the acceptance test when an unknown endpoint (i.e. not
stubbed) is used. We want to ensure that all API endpoints used in an
acceptance test are stubbed and do not otherwise silently fail with a
404.
The logs on failure output include a configuration that developers can
simply copy-paste to `test.toml` to stub the missing API endpoint. It'll
look something like:
```
[[Server]]
Pattern = "<method> <path>"
Response.Body = '''
<response body here>
'''
Response.StatusCode = <response status-code here>
```
## Tests
Manually:
output.txt when an endpoint is not found:
```
>>> [CLI] jobs create --json {"name":"abc"}
Error: No stub found for pattern: POST /api/2.1/jobs/create
```
How this renders in the test logs:
```
--- FAIL: TestAccept/workspace/jobs/create (0.03s)
server.go:46:
----------------------------------------
No stub found for pattern: POST /api/2.1/jobs/create
To stub a response for this request, you can add
the following to test.toml:
[[Server]]
Pattern = "POST /api/2.1/jobs/create"
Response.Body = '''
<response body here>
'''
Response.StatusCode = <response status-code here>
----------------------------------------
```
Manually checked that the debug mode still works.
## Changes
- Print warnings and errors by default.
- Fix ErrAlreadyPrinted not to be logged at Error level.
- Format log messages as "Warn: message" instead of "WARN" to make it
more readable and in-line with the rest of the output.
- Only print attributes (pid, mutator, etc) and time when the overall
level is debug (so --debug output has not changed much).
## Tests
- Existing acceptance tests show how warning messages appear in various
test case.
- Added new test for `--debug` output.
- Add sort_lines.py helper to avoid dependency on 'sort' which is
locale-sensitive.
## Changes
Extend testserver for bundle deployment:
- Allocate a new workspace per test case to isolate test cases from each
other
- Support jobs get/list/create
- Support creation and listing of workspace files
## Tests
Using existing acceptance tests
## Changes
HTTP headers like the User-Agent are an important part of our internal
ETL pipelines. This PR adds the ability to validate the headers used in
an HTTP request as part of our acceptance tests.
## Tests
Modifying existing test.
#2201 disabled using JSON as part of a bundle definition. I believe this
was not intended.
## Changes
Accept json files as includes, just as YAML files.
## Tests
Covered by the tests in #2201
## Changes
The APIs at Databricks when returning a non `200` status code will
return a response body of the format:
```
{
"error_code": "Error code",
"message": "Human-readable error message."
}
```
This PR adds the ability to stub non-200 status codes in the test
server, allowing us to mock API errors from Databricks.
## Tests
New test
$VARNAME is what we use for environment variables, it's good to
separate.
Some people use envsubst for homemade variable interpolation, it's also
good to have separation there.
## Changes
- Include acceptance directory in integration tests. Acceptance tests
will not start local server if CLOUD_ENV is set, so they become
integration tests.
- Add dependency for vendor to integration, so that CLI can be build
there.
- Implement LocalOnly option in test.toml to opt out of running
acceptance tests as integration tests. Use it in certain tests that are
difficult or not necessary to fix when run as integration tests.
- Update terraform test to redact timings out.
- Clean up .workspace.current_user from outputs of the tests.
## Tests
Existing tests.
## Changes
- Include compact relPath in the error message title. Include full paths
in separate lines below.
- Previously sometimes full paths were printed, sometime only rel path.
## Tests
Manually trigger the errors.
It is proven to be not necessary.
```
~/work/cli/acceptance % hyperfine -w 2 'go test' # with change:
Benchmark 1: go test
Time (mean ± σ): 4.983 s ± 0.209 s [User: 6.073 s, System: 9.869 s]
Range (min … max): 4.792 s … 5.483 s 10 runs
~/work/cli/acceptance % git stash # without change:
~/work/cli/acceptance % hyperfine -w 2 'go test'
Benchmark 1: go test
Time (mean ± σ): 5.018 s ± 0.100 s [User: 6.142 s, System: 10.234 s]
Range (min … max): 4.899 s … 5.182 s 10 runs
```
## Changes
- Add a script install_terraform.py that downloads terraform and
provider and generates a config to use, inspired by
https://gist.github.com/pietern/1cb6b6f3e0a452328e13cdc75031105e
- Make acceptance tests run this script once before running the tests
and set the required env vars to make cli use this terraform
installation.
- Use OS-specific directory for things that are build by acceptance test
runner (CLI and terraform).
This enables acceptance tests against cloud #2242 and local test for
bundle deploy #2254.
## Tests
- Add an acceptance test for standalone terraform. This is useful to
debug terraform with TF_LOG=DEBUG to see that it uses local provider.
- Other acceptance tests are updated with regard to terraform exec path.
- The overall time for tests locally is unchanged (if terraform is
already fetched).
## Changes
With this PR, any acceptance tests that define custom server stubs in
`test.toml` will automatically record all HTTP requests made and assert
on them.
Builds on top of https://github.com/databricks/cli/pull/2226
## Tests
Modifying existing acceptance test.
## Changes
This PR allows us to define custom server stubs in a `test.toml` file.
Note: A followup PR will add functionality to do assertions on the API
request itself.
## Tests
New acceptance test.
## Changes
This PR registers the `server.Close()` function to be run during test
cleanup in the server initialization function. This ensures that all
test servers are closed as soon as the test they are scoped to finish.
Motivated by https://github.com/databricks/cli/pull/2255/files where a
regression was introduced where we did not close the test server.
## Tests
N/A
## Changes
Added support for double underscore variable references.
Previously we made this restriction stronger with no particular reason,
TF provider supports multiple underscores and thus DABs should do as
well.
Fixes#1753
## Tests
Added acceptance and integration tests
## Changes
Noticed this when working on
https://github.com/databricks/cli/pull/2221. `<` is a special HTML
character that is encoded during text replacement when using
`AssertEqualTexts`.
## Tests
N/A
## Changes
- If CLOUD_ENV is set to do not override with dummy value. This allows
running acceptance tests as integration tests.
- Needed for https://github.com/databricks/cli/pull/2242
## Tests
Manually run the test suite against dogfood. `CLOUD_ENV=aws go test
./acceptance`
## Changes
- Do not start replacement / comparison if file is too large or not
valid utf-8.
- This helps to prevent replacements if there is accidentally a large
binary (e.g. terraform).
## Tests
Found this problem when working on
https://github.com/databricks/cli/pull/2242 -- the tests tried to
applied replacements on terraform binary and crashed. With this change,
an error is reported instead.
## Changes
- Replace development version with $DEV_VERSION
- Update experimental-jobs-as-code to make use of it.
## Tests
- Existing tests.
- Using this in https://github.com/databricks/cli/pull/2213
There is a speed up in 0.5s but it is still 4.4s, so something else is
slow there.
Benchmarking bundle/templates/experimental-jobs-as-code:
```
# Without UV_CACHE_DIR
~/work/cli/acceptance/bundle/templates/experimental-jobs-as-code % hyperfine --warmup 2 'testme -count=1'
Benchmark 1: testme -count=1
Time (mean ± σ): 4.950 s ± 0.079 s [User: 2.730 s, System: 8.524 s]
Range (min … max): 4.838 s … 5.076 s 10 runs
# With UV_CACHE_DIR
~/work/cli/acceptance/bundle/templates/experimental-jobs-as-code % hyperfine --warmup 2 'testme -count=1'
Benchmark 1: testme -count=1
Time (mean ± σ): 4.410 s ± 0.049 s [User: 2.669 s, System: 8.710 s]
Range (min … max): 4.324 s … 4.467 s 10 runs
```
## Changes
- Ability to extend a list of replacements via test.toml
- Modify selftest to both demo this feature and to get rid of sed on
Windows.
## Tests
Acceptance tests. I'm also using it
https://github.com/databricks/cli/pull/2213 for things like pid.
## Changes
- File comparison files in acceptance test, print the contents of all
applied replacements. Do it once per test.
- Remove duplicate entries in replacement list.
## Tests
Manually, change out files of existing test, you'll get this printed
once, after first assertion:
```
acceptance_test.go:307: Available replacements:
REPL /Users/denis\.bilenko/work/cli/acceptance/build/databricks => $$CLI
REPL /private/var/folders/5y/9kkdnjw91p11vsqwk0cvmk200000gp/T/TestAccept598522733/001 => $$TMPHOME
...
```
## Changes
- New test covering failures in reading .git. One case results in error,
some result in warning (not shown).
- New helper withdir runs commands in a subdirectory.
## Tests
New acceptance test.
## Changes
- Acceptance tests load test.toml to configure test behaviour.
- If file is not found in the test directory, parents are searched,
until the test root.
- Currently there is one option: runtime.GOOS to switch off tests per
OS.
## Tests
Using it in https://github.com/databricks/cli/pull/2223 to disable test
on Windows that cannot be run there.
## Changes
When adding path, a few things should take care of:
- symlink expansion
- forward/backward slashes, so that tests could do sed 's/\\\\/\//g' to
make it pass on Windows (see
acceptance/bundle/syncroot/dotdot-git/script)
SetPath() function takes care of both.
This PR uses SetPath() on all paths consistently.
## Tests
Existing tests.
## Changes
New source of default values for variables - variable file
`.databricks/bundle/<target>/variable-overrides.json`
CLI tries to stat and read that file every time during variable
initialisation phase
<!-- Summary of your changes that are easy to understand -->
## Tests
Acceptance tests
## Changes
`include` section is used only to include other bundle configuration
YAML files. If any other file type is used, raise an error and guide
users to use `sync.include` instead
## Tests
Added acceptance test
---------
Co-authored-by: Julia Crawford (Databricks) <julia.crawford@databricks.com>
## Changes
If there are unreadable files in a directory, raise an error but
continue with further diagnostics, because the answer is in the script
output.
## Tests
Manually - I'm working on some tests that create unreadable files, the
report is much better with this change.
## Changes
If git is not detected, set default worktree root to sync root.
Otherwise NewFileSet/View raise an error about worktree root being
outside view root in acceptance/bundle/sync-paths-dotdot.
This behavior is introduced in
https://github.com/databricks/cli/pull/1945
Stacked on https://github.com/databricks/cli/pull/2202
## Tests
Existing tests.
- Move acceptance/bundle/sync-paths-dotdot test to
acceptance/bundle/syncroot/dotdot-notgit
- Add new test acceptance/bundle/syncroot/dotdot-git
Fix replacer to work with this test and on Windows:
- Make PATH work on Windows by using EvalSymlinks.
- Make concatenated path match within JSON but stripping quotes.
## Changes
- If you pass -inprocess flag to acceptance tests, they will run in the
same process as test itself. This enables debugging.
- If you set singleTest variable on top of acceptance_test.go, you'll
only run that test and with inprocess mode. This is intended for
debugging in VSCode.
- (minor) Converted KeepTmp to flag -keeptmp from env var KEEP_TMP for
consistency with other flags.
## Tests
- I verified that acceptance tests pass with -inprocess mode: `go test
-inprocess < /dev/null | cat`
- I verified that debugging in VSCode works: set a test name in
singleTest variable, set breakpoints inside CLI and click "debug test"
in VSCode.
## Changes
- Add a new method Clone() on ReplacementContext
- Use it when passing common replacements to test cases.
## Tests
Manually. I have a different branch where this bug manifested and this
change helped.
Based on integration test from @andrewnester in #2194
Manually checked that this databricks.yml passes validation on v0.235.0
but fails on v0.236.0, very like it was broken in
https://github.com/databricks/cli/pull/1945
This also adds replacements for tmpdir, it's parent and (just in case)
grand parent.
## Changes
This is not needed because the command group is already returned by
`workspace.All()`.
The additional command registration was added in #1679.
## Tests
Acceptance test.
## Changes
Add experimental-jobs-as-code template allowing defining jobs using
Python instead of YAML through the `databricks-bundles` PyPI package.
## Tests
Manually and acceptance tests.
## Changes
Include a materialized copy of built-in templates as reference output.
This updates the output comparison logic to work against an output
directory. The `doComparison` function now always works on real files.
It can now tell apart non-existing files and empty files (e.g., the
`.gitkeep` files in templates).
## Changes
This includes a change to the defaults for the output directory flags of
the "generate" commands. These defaults included the expanded working
directory. This can be omitted because it is implied.
## Changes
The assertions on the output made are now captured in the `output.*`
files. These don't capture intent like actual assertions do, but we
still have regular test coverage in the path translation tests under
`bundle/config/mutator`.
## Tests
Tests pass.
## Changes
The materialized templates included in #2146 include Python code that we
require to be formatted. Instead of running ruff as part of the
testcase, we can enforce that all Python code in the repository is
formatted. It won't be possible to have a passing acceptance test for
template initialization with unformatted code.
## Changes
- Instead of doing 2 passes on variable resolution, do a loop until
there are no more updates (or we reach count 100).
- Stacked on top of #2163 which is a regression test for this:
acceptance/bundle/variables/complex-transitive-deep
## Tests
Existing tests, new regression tests.
These tests already passed before, added for completeness:
- acceptance/bundle/variables/cycle
- acceptance/bundle/variables/complex-cross-ref
Follow up to #2157. That PR repeated variable resolution. This test
still does not resolve fully but would resolve with 3 passes. This is
slightly different from complex-transitive-deeper - this test does not
show any errors, the issue is purely not enough passes.
## Changes
- Remove ResolveVariableReferencesInComplexVariables - it blocked
complex-within-complex for no good reason.
- Repeat regular resolution twice, it helps with a couple test cases we
have.
There may be a case for running it 3 times or more in a loop, but there
is no test case for that, so this PR is simple incremental improvement.
## Tests
Existing acceptance tests. Previously all unit tests for complex
variables were converted to acceptance tests, to capture this change and
ensure nothing breaks.
## Changes
It covers both https://$DATABRICKS_HOST and http://$DATABRICKS_HOST so
the test output does not change between local and the cloud.
## Tests
Existing tests using golden files (acceptance and integration) catch
this and were updated.
Anton Nekipelov
Andrew Nester
Ilya Kuznetsov
Denis Bilenko
Lennart Kats (databricks)
Pieter Noordhuis
shreyas-goenka
Gleb Kanterov
Simon Poltier