## Changes
This PR adds a warning which gives users clear guidance when they try to
use variable interpolation for an auth field.
## Tests
Modify existing acceptance test.
## Changes
Defining an include section in config files other than the main
`databricks.yml` file fails silently. With this PR users will get a
warning when they try this.
## Tests
Acceptance test.
## Changes
- Instead of constructing chains of mutators and then executing them,
execute them directly.
- Remove functionality related to chain-building: Seq, If, Defer,
newPhase, logString.
- Phases become functions that apply the changes directly rather than
construct mutator chains that will be called later.
- Add a helper ApplySeq to call multiple mutators, use it where
Apply+Seq were used before.
This is intended to be a refactoring without functional changes, but
there are a few behaviour changes:
- Since defer() is used to call unlock instead of bundle.Defer()
unlocking will now happen even in case of panics.
- In --debug, the phase names are are still logged once before start of
the phase but each entry no longer has 'seq' or phase name in it.
- The message "Deployment complete!" was printed even if
terraform.Apply() mutator had an error. It no longer does that.
## Motivation
The use of the chains was necessary when mutators were returning a list
of other mutators instead of calling them directly. But that has since
been removed, so now the chain machinery have no purpose anymore.
Use of direct functions simplifies the logic and makes bugs more
apparent and easy to fix.
Other improvements that this unlocks:
- Simpler stacktraces/debugging (breakpoints).
- Use of functions with narrowly scoped API: instead of mutators that
receive full bundle config, we can use focused functions that only deal
with sections they care about prepareGitSettings(currentGitSection) ->
updatedGitSection. This makes the data flow more apparent.
- Parallel computations across mutators (within phase): launch
goroutines fetching data from APIs at the beggining, process them once
they are ready.
## Tests
Existing tests.
## Changes
Instead of LocalOnly with non-composable semantics there are two
composable options:
- Local - enable test locally
- Cloud - enable test on the cloud
By default Cloud is switched off except in bundle (but not in
bundle/variables and bundle/help).
## Tests
Using this in #2383 to have test that runs on cloud but not locally.
## Changes
- Add 'serverless' prompt to default-python template (default is
currently set to "no").
- This is a simplified version of
https://github.com/databricks/cli/pull/2348 with 'auto' functionality
removed.
## Tests
- Split default-python into default-python/classic,
default-python/serverless, default-python/serverless-customcatalog.
- Manually check that "bundle init default-python" with serverless=yes
can be deployed and run on dogfood and test env.
## Changes
Now when `profile` flag is used we won't pick up host from bundle
anymore and use the one provided by -p flag
Previous behaviour in the context of bundle
```
databricks current-user me -p profile_name
Error: cannot resolve bundle auth configuration: config host mismatch: profile uses host https://non-existing-subdomain.databricks.com, but CLI configured to use https://foo.com
```
New behaviour (make an api call)
```
databricks current-user me -p profile_name
{
email: "foo@bar.com"
...
}
```
We still load bundle configuration when `-t` flag provide because we
want to load host information from the target.
Fixes#1358
## Tests
Added acceptance test
## Changes
1. Change the **default-python** bundle template to set
`data_security_mode` of a cluster to SINGLE_USER
2. Change the **experimental-jobs-as-code** bundle template to set
`data_security_mode` of a cluster to SINGLE_USER
## Why
Explicitly adding this field saves experienced users from confusion onto
what security mode is applied to the cluster
## Tests
Changed existing unit and integration tests to pass with this change
## Changes
Added PyPi and Maven libraries tests
Needed for this PR since we don't currently have any coverage for PyPi
or Maven libraries
https://github.com/databricks/cli/pull/2382
## Changes
Previously using python wheel tasks in the tasks with compute referering
to interactive cluster defied in the same bundle would produce a warning
like below
```
GET /api/2.1/clusters/get?cluster_id=${resources.clusters.development_cluster.id}
< HTTP/2.0 400 Bad Request
< {
< "error_code": "INVALID_PARAMETER_VALUE",
< "message": "Cluster ${resources.clusters.development_cluster.id} does not exist"
< } pid=14465 mutator=seq mutator=initialize mutator=seq mutator=PythonWrapperWarning sdk=true
```
This PR fixes it by making sure that we check spark version for such
clusters based on its bundle configuration and don't make API calls
## Tests
Added acceptance test
## Changes
CLI generation template was using RequiredPathField from incorrect
request entity (body field from request and not request itself). Thus
for some of the commands required path parameters were not required when
--json was specified.
## Tests
Regenerated commands work correctly
## Changes
Added missing .gitignore files to templates
## Tests
There were some incorrect snapshots of gitignore files in acceptance
tests, probably generated by testing infra. Updated them to new files
---------
Co-authored-by: Lennart Kats (databricks) <lennart.kats@databricks.com>
## Changes
Previously, one could not set `LocalOnly=true` in parent directory and
then override it with `LocalOnly=false` in child directory. This is
because, `false` is considered empty value by mergo.
In order to distinguish between 'explicitly set to false' and 'not set',
I've changed all simple variables in config to be pointers. Now, one can
always override those, because non-nil pointers are not null (with
mergo.WithoutDereference option).
## Tests
Manually:
```
~/work/cli/acceptance/bundle/templates/default-python % cat test.toml # add this new file
LocalOnly = false
~/work/cli/acceptance/bundle/templates/default-python % CLOUD_ENV=aws go test ../../.. -run ^TestAccept$/^bundle$/^templates$/^default-python$ -v
(the test is no longer skipped)
```
## Changes
- Instead of collecting requests in memory and writing them at the end
of the test, write them right away. Then test authors can do filtering
with jq in 'script' or collect individual files per different command.
- testserver is now simpler - it just calls a caller-provided function.
The logging logic is moved to acceptance_test.go.
See https://github.com/databricks/cli/pull/2359/files#r1967591173
## Tests
Existing tests.
## Changes
This removes the `run-as` property from the default templates. It's a
useful property but it still only works for jobs and it makes the
default databricks.yml a bit longer. It seems like users can just learn
about it from the docs and/or vary their deployment identity.
Depends on https://github.com/databricks/cli/pull/1712.
Shreyas Goenka
Andrew Nester
shreyas-goenka
Denis Bilenko
Anton Nekipelov
Ilya Kuznetsov
Lennart Kats (databricks)