## Changes
If git is not detected, set default worktree root to sync root.
Otherwise NewFileSet/View raise an error about worktree root being
outside view root in acceptance/bundle/sync-paths-dotdot.
This behavior is introduced in
https://github.com/databricks/cli/pull/1945
Stacked on https://github.com/databricks/cli/pull/2202
## Tests
Existing tests.
## Changes
It is possible to pass variable to enum fields but json-schema doesn't
accept it. This PR adds `oneOf` for enum types that includes `${var-*}`
pattern
## Tests
Manually checked in VSCode
- Move acceptance/bundle/sync-paths-dotdot test to
acceptance/bundle/syncroot/dotdot-notgit
- Add new test acceptance/bundle/syncroot/dotdot-git
Fix replacer to work with this test and on Windows:
- Make PATH work on Windows by using EvalSymlinks.
- Make concatenated path match within JSON but stripping quotes.
## Changes
- If you pass -inprocess flag to acceptance tests, they will run in the
same process as test itself. This enables debugging.
- If you set singleTest variable on top of acceptance_test.go, you'll
only run that test and with inprocess mode. This is intended for
debugging in VSCode.
- (minor) Converted KeepTmp to flag -keeptmp from env var KEEP_TMP for
consistency with other flags.
## Tests
- I verified that acceptance tests pass with -inprocess mode: `go test
-inprocess < /dev/null | cat`
- I verified that debugging in VSCode works: set a test name in
singleTest variable, set breakpoints inside CLI and click "debug test"
in VSCode.
## Changes
- Add a new method Clone() on ReplacementContext
- Use it when passing common replacements to test cases.
## Tests
Manually. I have a different branch where this bug manifested and this
change helped.
Based on integration test from @andrewnester in #2194
Manually checked that this databricks.yml passes validation on v0.235.0
but fails on v0.236.0, very like it was broken in
https://github.com/databricks/cli/pull/1945
This also adds replacements for tmpdir, it's parent and (just in case)
grand parent.
## Changes
The SelectTarget mutator (part of Load phase) clears bundle.git.inferred
flag but it is not set until later - Initialize phase / LoadGitDetails
mutator.
## Tests
Existing tests.
## Changes
This is not needed because the command group is already returned by
`workspace.All()`.
The additional command registration was added in #1679.
## Tests
Acceptance test.
Combine 'make cover' and 'make acc-cover' into single command. They
still write coverage into different files -- it would be useful to see
separate coverage numbers.
Note, we're not making use of coverage information yet. However, running
tests in CI with coverage will
- let us catch issues that only manifest when coverage is enabled, like
https://github.com/databricks/cli/pull/2150
- will let us know if there are any issues with running coverage on CI
before investing in additional coverage support
## Summary of changes
This PR introduces three new abstractions:
1. `Resolver`: Resolves which reader and writer to use for a template.
2. `Writer`: Writes a template project to disk. Prompts the user if
necessary.
3. `Reader`: Reads a template specification from disk, built into the
CLI or from GitHub.
Introducing these abstractions helps decouple reading a template from
writing it. When I tried adding telemetry for the `bundle init` command,
I noticed that the code in `cmd/init.go` was getting convoluted and hard
to test. A future change could have accidentally logged PII when a user
initialised a custom template.
Hedging against that risk is important here because we use a generic
untyped `map<string, string>` representation in the backend to log
telemetry for the `databricks bundle init`. Otherwise, we risk
accidentally breaking our compliance with our centralization
requirements.
### Details
After this PR there are two classes of templates that can be
initialized:
1. A `databricks` template: This could be a builtin template or a
template outside the CLI like mlops-stacks, which is still owned and
managed by Databricks. These templates log their telemetry arguments and
template name.
2. A `custom` template: These are templates created by and managed by
the end user. In these templates we do not log the template name and
args. Instead a generic placeholder string of "custom" is logged in our
telemetry system.
NOTE: The functionality of the `databricks bundle init` command remains
the same after this PR. Only the internal abstractions used are changed.
## Tests
New unit tests. Existing golden and unit tests. Also a fair bit of
manual testing.
## Changes
Add experimental-jobs-as-code template allowing defining jobs using
Python instead of YAML through the `databricks-bundles` PyPI package.
## Tests
Manually and acceptance tests.
## Changes
Setting Verbose=false on testcli.Runner disables all logging related to
running process (stdout, stderr, error, args).
I'm using this in #2184 where I'm using testcli runner to run acceptance
tests and seeing all output is not useful.
## Tests
Manually inspecting test output in #2184
## Changes
- Fix incorrect use Errorf on literal string. This resulted in garbage
output in tests diagnostics where % was replaced by "(MISSING)".
- Enable linter on testingT.Errorf.
Note, the autofix by the linter is wrong, it proposes `t.Errorf("%s",
string)` but it should be `t.Error(string)`. That can corrected manually
though.
## Tests
Linter was tested manually by reverting the fix on Errorf.
## Changes
Include a materialized copy of built-in templates as reference output.
This updates the output comparison logic to work against an output
directory. The `doComparison` function now always works on real files.
It can now tell apart non-existing files and empty files (e.g., the
`.gitkeep` files in templates).
## Changes
This includes a change to the defaults for the output directory flags of
the "generate" commands. These defaults included the expanded working
directory. This can be omitted because it is implied.
## Changes
The assertions on the output made are now captured in the `output.*`
files. These don't capture intent like actual assertions do, but we
still have regular test coverage in the path translation tests under
`bundle/config/mutator`.
## Tests
Tests pass.
## Changes
This came up in #2122 where relative library paths showed up with
backslashes on Windows. It's hard to run acceptance tests where paths
may be in either form. This change updates path translation logic to
always use forward slash-separated paths, including for absolute paths.
## Tests
* Unit tests pass.
* Confirmed that code where library paths are used uses the `filepath`
package for path manipulation. The functions in this package always
normalize their inputs to be platform-native paths.
* Confirmed that code that uses absolute paths works with forward
slash-separated paths on Windows.
## Changes
The materialized templates included in #2146 include Python code that we
require to be formatted. Instead of running ruff as part of the
testcase, we can enforce that all Python code in the repository is
formatted. It won't be possible to have a passing acceptance test for
template initialization with unformatted code.
## Changes
- Instead of doing 2 passes on variable resolution, do a loop until
there are no more updates (or we reach count 100).
- Stacked on top of #2163 which is a regression test for this:
acceptance/bundle/variables/complex-transitive-deep
## Tests
Existing tests, new regression tests.
These tests already passed before, added for completeness:
- acceptance/bundle/variables/cycle
- acceptance/bundle/variables/complex-cross-ref
## Changes
Fixes https://github.com/databricks/cli/issues/1977.
This PR modifies the bundle configuration to capture the dependency that
a UC Volume or a DLT pipeline might have on a UC schema at deployment
time. It does so by replacing the schema name with a reference of the
form `${resources.schemas.foo.name}`.
For example:
The following UC Volume definition depends on the UC schema with the
name `schema_name`. This mutator converts this configuration
from:
```
resources:
volumes:
bar:
catalog_name: catalog_name
name: volume_name
schema_name: schema_name
schemas:
foo:
catalog_name: catalog_name
name: schema_name
```
to:
```
resources:
volumes:
bar:
catalog_name: catalog_name
name: volume_name
schema_name: ${resources.schemas.foo.name}`
schemas:
foo:
catalog_name: catalog_name
name: schema_name
```
## Tests
Unit tests and manually.
Follow up to #2157. That PR repeated variable resolution. This test
still does not resolve fully but would resolve with 3 passes. This is
slightly different from complex-transitive-deeper - this test does not
show any errors, the issue is purely not enough passes.
## Changes
When users create one or more Databricks apps in their bundle it can
lead to initial bundle deployment being slower because apps need to wait
until their compute is fully provisioned and started.
This PR adds a message to warn about it. This message will be removed
when `no_compute` option becomes available in TF provider and used in
DABs (https://github.com/databricks/cli/pull/2144)
## Changes
Pipeline backend requires `target` to be always specified.
In order to do this we will create an unique schema as part of
`TestBundlePipelineRecreateWithoutAutoApprove` test which will be used
in pipelines
## Tests
```
helpers_test.go:148: stderr: Destroy complete!
--- PASS: TestBundlePipelineRecreateWithoutAutoApprove (415.39s)
PASS
coverage: [no statements]
ok github.com/databricks/cli/integration/bundle 416.141s coverage: [no statements]
```
Shreyas Goenka
shreyas-goenka
Denis Bilenko
Ilya Kuznetsov
Pieter Noordhuis
Gleb Kanterov
Andrew Nester