## Changes
Noticed this when working on
https://github.com/databricks/cli/pull/2221. `<` is a special HTML
character that is encoded during text replacement when using
`AssertEqualTexts`.
## Tests
N/A
## Changes
- If CLOUD_ENV is set to do not override with dummy value. This allows
running acceptance tests as integration tests.
- Needed for https://github.com/databricks/cli/pull/2242
## Tests
Manually run the test suite against dogfood. `CLOUD_ENV=aws go test
./acceptance`
## Changes
- Do not start replacement / comparison if file is too large or not
valid utf-8.
- This helps to prevent replacements if there is accidentally a large
binary (e.g. terraform).
## Tests
Found this problem when working on
https://github.com/databricks/cli/pull/2242 -- the tests tried to
applied replacements on terraform binary and crashed. With this change,
an error is reported instead.
## Changes
- Replace development version with $DEV_VERSION
- Update experimental-jobs-as-code to make use of it.
## Tests
- Existing tests.
- Using this in https://github.com/databricks/cli/pull/2213
There is a speed up in 0.5s but it is still 4.4s, so something else is
slow there.
Benchmarking bundle/templates/experimental-jobs-as-code:
```
# Without UV_CACHE_DIR
~/work/cli/acceptance/bundle/templates/experimental-jobs-as-code % hyperfine --warmup 2 'testme -count=1'
Benchmark 1: testme -count=1
Time (mean ± σ): 4.950 s ± 0.079 s [User: 2.730 s, System: 8.524 s]
Range (min … max): 4.838 s … 5.076 s 10 runs
# With UV_CACHE_DIR
~/work/cli/acceptance/bundle/templates/experimental-jobs-as-code % hyperfine --warmup 2 'testme -count=1'
Benchmark 1: testme -count=1
Time (mean ± σ): 4.410 s ± 0.049 s [User: 2.669 s, System: 8.710 s]
Range (min … max): 4.324 s … 4.467 s 10 runs
```
## Changes
- Remove DetectInterpreters from DetectExecutable call: python3 or
python should always be on on the PATH. We don't need to detect
non-standard situations like python3.10 is present but python3 is not.
- I moved DetectInterpreters to cmd/labs where it is still used.
This is a follow up to https://github.com/databricks/cli/pull/2034
## Tests
Existing tests.
## Changes
- Ability to extend a list of replacements via test.toml
- Modify selftest to both demo this feature and to get rid of sed on
Windows.
## Tests
Acceptance tests. I'm also using it
https://github.com/databricks/cli/pull/2213 for things like pid.
## Changes
- File comparison files in acceptance test, print the contents of all
applied replacements. Do it once per test.
- Remove duplicate entries in replacement list.
## Tests
Manually, change out files of existing test, you'll get this printed
once, after first assertion:
```
acceptance_test.go:307: Available replacements:
REPL /Users/denis\.bilenko/work/cli/acceptance/build/databricks => $$CLI
REPL /private/var/folders/5y/9kkdnjw91p11vsqwk0cvmk200000gp/T/TestAccept598522733/001 => $$TMPHOME
...
```
## Changes
- New test covering failures in reading .git. One case results in error,
some result in warning (not shown).
- New helper withdir runs commands in a subdirectory.
## Tests
New acceptance test.
## Changes
For the most recent release, I had to re-run the "publish-winget" action
a couple of times before it passed. The underlying issue that causes the
failure should be solved by the latest version of the action, but upon
inspection of the latest version, I found that it always installs the
latest version of [Komac](https://github.com/russellbanks/Komac). To
both fix the issue and lock this down further, I updated our action to
call Komac directly instead of relying on a separate action to do this
for us.
## Tests
Successful run in
https://github.com/databricks/cli/actions/runs/12951529979.
## Changes
- Acceptance tests load test.toml to configure test behaviour.
- If file is not found in the test directory, parents are searched,
until the test root.
- Currently there is one option: runtime.GOOS to switch off tests per
OS.
## Tests
Using it in https://github.com/databricks/cli/pull/2223 to disable test
on Windows that cannot be run there.
## Changes
When adding path, a few things should take care of:
- symlink expansion
- forward/backward slashes, so that tests could do sed 's/\\\\/\//g' to
make it pass on Windows (see
acceptance/bundle/syncroot/dotdot-git/script)
SetPath() function takes care of both.
This PR uses SetPath() on all paths consistently.
## Tests
Existing tests.
## Changes
New source of default values for variables - variable file
`.databricks/bundle/<target>/variable-overrides.json`
CLI tries to stat and read that file every time during variable
initialisation phase
<!-- Summary of your changes that are easy to understand -->
## Tests
Acceptance tests
## Changes
`include` section is used only to include other bundle configuration
YAML files. If any other file type is used, raise an error and guide
users to use `sync.include` instead
## Tests
Added acceptance test
---------
Co-authored-by: Julia Crawford (Databricks) <julia.crawford@databricks.com>
## Changes
- Propagate env vars USE_SDK_V2_RESOURCES and $USE_SDK_V2_DATA_SOURCES
to terraform
- This are troubleshooting helpers for resources migrated to new plugin
framework, recommended here:
https://registry.terraform.io/providers/databricks/databricks/latest/docs/guides/troubleshooting#plugin-framework-migration-problems
- This current unblocks deploying quality monitors, see
https://github.com/databricks/terraform-provider-databricks/issues/4229#issuecomment-2520344690
## Tests
Manually testing that I can deploy quality monitor after this change
with `USE_SDK_V2_RESOURCES="databricks_quality_monitor"` set
### Main branch:
```
~/work/databricks_quality_monitor_repro % USE_SDK_V2_RESOURCES="databricks_quality_monitor" ../cli/cli-main bundle deploy
Uploading bundle files to /Workspace/Users/denis.bilenko@databricks.com/.bundle/quality_monitor_bundle/default/files...
Deploying resources...
Updating deployment state...
Deployment complete!
Error: terraform apply: exit status 1
Error: Provider produced inconsistent result after apply
When applying changes to databricks_quality_monitor.monitor_trips, provider
"provider[\"registry.terraform.io/databricks/databricks\"]" produced an
unexpected new value: .data_classification_config: block count changed from 0
to 1.
This is a bug in the provider, which should be reported in the provider's own
issue tracker.
```
### This branch:
```
~/work/databricks_quality_monitor_repro % USE_SDK_V2_RESOURCES="databricks_quality_monitor" ../cli/cli bundle deploy
Uploading bundle files to /Workspace/Users/denis.bilenko@databricks.com/.bundle/quality_monitor_bundle/default/files...
Deploying resources...
Updating deployment state...
Deployment complete!
```
### Config:
```
~/work/databricks_quality_monitor_repro % cat databricks.yml
bundle:
name: quality_monitor_bundle
resources:
quality_monitors:
monitor_trips:
table_name: main.denis-bilenko-cuj-pe34.trips_sanitized_1
output_schema_name: main.denis-bilenko-cuj-pe34
assets_dir: /Workspace/Users/${workspace.current_user.userName}/quality_monitor_issue
snapshot: {}
```
## Changes
If there are unreadable files in a directory, raise an error but
continue with further diagnostics, because the answer is in the script
output.
## Tests
Manually - I'm working on some tests that create unreadable files, the
report is much better with this change.
## Changes
Allows custom untyped fields in the root config in json-schema so it
doesn't highlight errors when using yaml-anchors.
Example use case:
```
tags: &job-tags
environment: ${bundle.target}
resources:
jobs:
db1:
tags:
<<: *job-tags
db1:
tags:
<<: *job-tags
```
One downside is that we don't highlight any unknown top-level properties
anymore (but they will still fail during CLI validation)
## Tests
Manually checked behavior in VSCode - it doesn't show validation error.
Also checked that other typed properties are still suggested
## Changes
Add a mechanism to load Python source locations in the Python mutator.
Previously, locations pointed to generated YAML. Now, they point to
Python sources instead. Python process outputs "locations.json"
containing locations of bundle paths, examples:
```json
{"path": "resources.jobs.job_0", "file": "resources/job_0.py", "line": 3, "column": 5}
{"path": "resources.jobs.job_0.tasks[0].task_key", "file": "resources/job_0.py", "line": 10, "column": 5}
{"path": "resources.jobs.job_1", "file": "resources/job_1.py", "line": 5, "column": 7}
```
Such locations form a tree, and we assign locations of the closest
ancestor to each `dyn.Value` based on its path. For example,
`resources.jobs.job_0.tasks[0].task_key` is located at `job_0.py:10:5`
and `resources.jobs.job_0.tasks[0].email_notifications` is located at
`job_0.py:3:5`, because we use the location of the job as the most
precise approximation.
This feature is only enabled if `experimental/python` is used.
Note: for now, we don't update locations with relative paths, because it
has a side effect in changing how these paths are resolved
## Example
```
% databricks bundle validate
Warning: job_cluster_key abc is not defined
at resources.jobs.examples.tasks[0].job_cluster_key
in resources/example.py:10:1
```
## Tests
Unit tests and manually
## Changes
If git is not detected, set default worktree root to sync root.
Otherwise NewFileSet/View raise an error about worktree root being
outside view root in acceptance/bundle/sync-paths-dotdot.
This behavior is introduced in
https://github.com/databricks/cli/pull/1945
Stacked on https://github.com/databricks/cli/pull/2202
## Tests
Existing tests.
## Changes
It is possible to pass variable to enum fields but json-schema doesn't
accept it. This PR adds `oneOf` for enum types that includes `${var-*}`
pattern
## Tests
Manually checked in VSCode
- Move acceptance/bundle/sync-paths-dotdot test to
acceptance/bundle/syncroot/dotdot-notgit
- Add new test acceptance/bundle/syncroot/dotdot-git
Fix replacer to work with this test and on Windows:
- Make PATH work on Windows by using EvalSymlinks.
- Make concatenated path match within JSON but stripping quotes.
## Changes
- If you pass -inprocess flag to acceptance tests, they will run in the
same process as test itself. This enables debugging.
- If you set singleTest variable on top of acceptance_test.go, you'll
only run that test and with inprocess mode. This is intended for
debugging in VSCode.
- (minor) Converted KeepTmp to flag -keeptmp from env var KEEP_TMP for
consistency with other flags.
## Tests
- I verified that acceptance tests pass with -inprocess mode: `go test
-inprocess < /dev/null | cat`
- I verified that debugging in VSCode works: set a test name in
singleTest variable, set breakpoints inside CLI and click "debug test"
in VSCode.
## Changes
- Add a new method Clone() on ReplacementContext
- Use it when passing common replacements to test cases.
## Tests
Manually. I have a different branch where this bug manifested and this
change helped.
Ilya Kuznetsov
Andrew Nester
Denis Bilenko
shreyas-goenka
dependabot[bot]
Pieter Noordhuis
Gleb Kanterov