## Why
The command execution ID is a UUID that should be initialized once, and
then have a consistent read value for the duration of the command.
As we start emitting telemetry events, we also want to emit the command
execution ID in the telemetry payload. This library helps consolidate
all reads to the `cmd-exec-id` value, and ensures there's only one write
for the UUID.
relevant comment:
https://github.com/databricks/cli/pull/2432#discussion_r1982884235
## Tests
Unit tests.
## Changes
<!-- Summary of your changes that are easy to understand -->
This PR makes changes to the labs code base to allow for offline
installation of labs projects (like UCX). By passing a flag
--offline=true, the code will skip checking for project versions and
download code from GitHub and instead will look from the local
installation folder. This cmd is useful in systems where there is
internet restriction, the user should follow a set-up as follows:
- install a labs project on a machine which has internet
- zip and copy the file to the intended machine and
- run databricks labs install <project name>--offline=true
it will look for the code in the same install directory and if present
load from there.
Closes#1646
related to https://github.com/databrickslabs/ucx/issues/3418
## Tests
<!-- How is this tested? -->
Added unit test case and tested.
NO_CHANGELOG=true
---------
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: Pieter Noordhuis <pieter.noordhuis@databricks.com>
Co-authored-by: Lennart Kats (databricks) <lennart.kats@databricks.com>
Co-authored-by: Denis Bilenko <denis.bilenko@databricks.com>
Co-authored-by: Julia Crawford (Databricks) <julia.crawford@databricks.com>
Co-authored-by: Ilya Kuznetsov <ilya.kuznetsov@databricks.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Andrew Nester <andrew.nester@databricks.com>
Co-authored-by: Anton Nekipelov <226657+anton-107@users.noreply.github.com>
Co-authored-by: shreyas-goenka <88374338+shreyas-goenka@users.noreply.github.com>
## Changes
Fixes#1888
## Why
`databricks queries list` is using override to define a template to
display a list in a nice table form. This Go template was referencing to
non existing anymore fields and hence failing.
## Tests
Added acceptance test
## Changes
Move the WorkspaceClient reader and setter from the root package to
`libs/command`.
## Why
This allows us to read the workspace client set in the context in all
CLI packages. This was not possible before because using
`root.WorkspaceClient` would often result in an import cycle.
This would also allow us to standardise reads of the workspace client to
be from the context in bundle commands. Today those are read from the
bundle object tree instead.
This is a natural followup to
https://github.com/databricks/cli/pull/2440.
## Tests
Existing tests and one new unit test.
NO_CHANGELOG=true
## Changes
This PR moves `ConfigUsed` from the root package to `libs/command`.
## Why
Having the ConfigUsed function in the root package is a problem because
that means we cannot use that function from outside the `root` package
since doing so often leads to an import cycle (because `root` imports
everything implicitly).
Moving it to a separate package that consolidates the read interaction
and solves the import cycle issue. Example where this would have
simplified code:
https://github.com/databricks/cli/pull/2432#discussion_r1983368092
I'd like to send PRs to do the same for the workspace client and account
client as well. I'll wait however until this one is merged incase people
have concerns about the approach here.
## Tests
Existing tests.
## Changes
- New library to patch existing whl with dynamic version. The version is
based on mtime of original wheel.
- New hidden command "selftest patchwhl" (for acceptance test).
## Why
Part of the project to deprecate dynamic version in setup.py template
(#1034). This would enable switching default-python to pyproject.toml
without dependency on setuptools-specific sections.
The dynamic version is needed to ensure that wheels are picked up by the
users code when doing development.
## Tests
- New unit tests that use uv to create envs on various versions and
install (and re-install) patched wheels there.
- New acceptance test that patches prebuilt whl and records the result
fully unzipped.
- New acceptance test helper setmtime.py to set mtime with nanosecond
precision cross-platform.
## Changes
- Added `service-principal-secrets` command
- Added `budget-policy-id` for apps
- `experiments.log-inputs` now requires `ID` parameter as an input
- Added `genie.get-space` command
- Added `providers.list-provider-share-assets` command
For the whole list of SDK changes see:
https://github.com/databricks/databricks-sdk-go/releases/tag/v0.59.0
## Changes
- Remove bundle.Parallel & bundle.ReadOnlyBundle.
- Add bundle.ApplyParallel, as a helper to migrate from bundle.Parallel.
- Keep ReadOnlyMutator as a separate type but it's now a subtype of
Mutator so it works on regular *Bundle. Having it as a separate type
prevents non-readonly mutators being passed to ApplyParallel
- validate.Validate becomes a function (was Mutator).
## Why
This a follow up to #2390 where we removed most of the tools to
construct chains of mutators. Same motivation applies here.
When it comes to read-only bundles, it's a leaky abstraction -- since
it's a shallow copy, it does not actually guarantee or enforce readonly
access to bundle. A better approach would be to run parallel operations
on independent narrowly-focused deep-copied structs, with just enough
information to carry out the task (this is not implemented here, but the
eventual goal). Now that we can just write regular code in phases and
not limited to mutator interface, we can switch to that approach.
## Tests
Existing tests.
---------
Co-authored-by: shreyas-goenka <88374338+shreyas-goenka@users.noreply.github.com>
## Changes
- Instead of constructing chains of mutators and then executing them,
execute them directly.
- Remove functionality related to chain-building: Seq, If, Defer,
newPhase, logString.
- Phases become functions that apply the changes directly rather than
construct mutator chains that will be called later.
- Add a helper ApplySeq to call multiple mutators, use it where
Apply+Seq were used before.
This is intended to be a refactoring without functional changes, but
there are a few behaviour changes:
- Since defer() is used to call unlock instead of bundle.Defer()
unlocking will now happen even in case of panics.
- In --debug, the phase names are are still logged once before start of
the phase but each entry no longer has 'seq' or phase name in it.
- The message "Deployment complete!" was printed even if
terraform.Apply() mutator had an error. It no longer does that.
## Motivation
The use of the chains was necessary when mutators were returning a list
of other mutators instead of calling them directly. But that has since
been removed, so now the chain machinery have no purpose anymore.
Use of direct functions simplifies the logic and makes bugs more
apparent and easy to fix.
Other improvements that this unlocks:
- Simpler stacktraces/debugging (breakpoints).
- Use of functions with narrowly scoped API: instead of mutators that
receive full bundle config, we can use focused functions that only deal
with sections they care about prepareGitSettings(currentGitSection) ->
updatedGitSection. This makes the data flow more apparent.
- Parallel computations across mutators (within phase): launch
goroutines fetching data from APIs at the beggining, process them once
they are ready.
## Tests
Existing tests.
## Changes
Now when `profile` flag is used we won't pick up host from bundle
anymore and use the one provided by -p flag
Previous behaviour in the context of bundle
```
databricks current-user me -p profile_name
Error: cannot resolve bundle auth configuration: config host mismatch: profile uses host https://non-existing-subdomain.databricks.com, but CLI configured to use https://foo.com
```
New behaviour (make an api call)
```
databricks current-user me -p profile_name
{
email: "foo@bar.com"
...
}
```
We still load bundle configuration when `-t` flag provide because we
want to load host information from the target.
Fixes#1358
## Tests
Added acceptance test
## Changes
CLI generation template was using RequiredPathField from incorrect
request entity (body field from request and not request itself). Thus
for some of the commands required path parameters were not required when
--json was specified.
## Tests
Regenerated commands work correctly
## Changes
- Insert newline after rendering indented JSON in bundle
validate/summary/run.
- This prevents "No newline at end of file" message in various cases,
for example when switching between recording raw output of the command
to output processed by jq, since jq does add a newline or when running
diff in acceptance tests.
## Tests
Manually running validate:
```
~/work/dabs_cuj_brickfood % ../cli/cli-main bundle validate -o json | tail -n 2 # without change
Error: root_path must start with '~/' or contain the current username to ensure uniqueness when using 'mode: development'
}
}%
~/work/dabs_cuj_brickfood % ../cli/cli bundle validate -o json | tail -n 2 # with change
Error: root_path must start with '~/' or contain the current username to ensure uniqueness when using 'mode: development'
}
}
~/work/dabs_cuj_brickfood %
```
Via #2316 -- see cleaner output there.
## Changes
This will generate bundle YAML configuration for Git based jobs but
won't download any related files as they are in Git repo.
Fixes#1423
## Tests
Added unit test
---------
Co-authored-by: Pieter Noordhuis <pieter.noordhuis@databricks.com>
## Changes
`CreatePipeline` is a more complete structure (superset of PipelineSpec
one) which enables support of additional fields such as `run_as` and
`allow_duplicate_names` in DABs configuration. Note: these fields are
subject to support in TF in order to correctly work.
## Tests
Existing tests pass + no fields are removed from JSON schema
Shreyas Goenka
shreyas-goenka
Hari Selvarajan
Andrew Nester
Denis Bilenko
dependabot[bot]