## Changes
`CreatePipeline` is a more complete structure (superset of PipelineSpec
one) which enables support of additional fields such as `run_as` and
`allow_duplicate_names` in DABs configuration. Note: these fields are
subject to support in TF in order to correctly work.
## Tests
Existing tests pass + no fields are removed from JSON schema
## Changes
Instead of using leaf-most config, all configs from root at
acceptance/test.toml to all intermediate ones to leaf config are merged
into one. Maps are merged, slices are appended, other values are
overridden.
I had to disable caching, because it is tricky when merging is involved
- deep copy is needed. There is performance
impact but currently it is tiny, about 1%.
Also, remove empty root config.
## Tests
Manually checked that inheritance of LocalOnly setting worked for these
tests:
Before - integration tests showed:
```
PASS acceptance.TestAccept/bundle/templates/wrong-url (0.70s)
PASS acceptance.TestAccept/bundle/templates/wrong-path (0.44s)
```
After:
```
SKIP acceptance.TestAccept/bundle/templates/wrong-url (0.00s)
SKIP acceptance.TestAccept/bundle/templates/wrong-path (0.00s)
acceptance_test.go:216: Disabled via LocalOnly setting in bundle/templates/test.toml, bundle/templates/wrong-path/test.toml (CLOUD_ENV=***)
```
## Changes
This PR fails the acceptance test when an unknown endpoint (i.e. not
stubbed) is used. We want to ensure that all API endpoints used in an
acceptance test are stubbed and do not otherwise silently fail with a
404.
The logs on failure output include a configuration that developers can
simply copy-paste to `test.toml` to stub the missing API endpoint. It'll
look something like:
```
[[Server]]
Pattern = "<method> <path>"
Response.Body = '''
<response body here>
'''
Response.StatusCode = <response status-code here>
```
## Tests
Manually:
output.txt when an endpoint is not found:
```
>>> [CLI] jobs create --json {"name":"abc"}
Error: No stub found for pattern: POST /api/2.1/jobs/create
```
How this renders in the test logs:
```
--- FAIL: TestAccept/workspace/jobs/create (0.03s)
server.go:46:
----------------------------------------
No stub found for pattern: POST /api/2.1/jobs/create
To stub a response for this request, you can add
the following to test.toml:
[[Server]]
Pattern = "POST /api/2.1/jobs/create"
Response.Body = '''
<response body here>
'''
Response.StatusCode = <response status-code here>
----------------------------------------
```
Manually checked that the debug mode still works.
## Changes
The intent of this PR is to clarify that the `databricks auth token`
command is not supported for M2M auth. Fixes:
https://github.com/databricks/cli/issues/1939
## Tests
Manually.
## Changes
- Print warnings and errors by default.
- Fix ErrAlreadyPrinted not to be logged at Error level.
- Format log messages as "Warn: message" instead of "WARN" to make it
more readable and in-line with the rest of the output.
- Only print attributes (pid, mutator, etc) and time when the overall
level is debug (so --debug output has not changed much).
## Tests
- Existing acceptance tests show how warning messages appear in various
test case.
- Added new test for `--debug` output.
- Add sort_lines.py helper to avoid dependency on 'sort' which is
locale-sensitive.
## Changes
Extend testserver for bundle deployment:
- Allocate a new workspace per test case to isolate test cases from each
other
- Support jobs get/list/create
- Support creation and listing of workspace files
## Tests
Using existing acceptance tests
## Changes
1. Allow `any` examples in json-schema type since we have many of them
in open api spec
2. Fix issue with missing overrides annotations when re-generating the
schema
## Tests
<!-- How is this tested? -->
## Changes
Added support for extras / optional Python dependencies in the labs CLI.
Added new `extras` field under install.
Example:
```yaml
install:
script: install.py
extras: cli
```
Resolves: #2257
## Tests
Manual test
## Changes
HTTP headers like the User-Agent are an important part of our internal
ETL pipelines. This PR adds the ability to validate the headers used in
an HTTP request as part of our acceptance tests.
## Tests
Modifying existing test.
#2201 disabled using JSON as part of a bundle definition. I believe this
was not intended.
## Changes
Accept json files as includes, just as YAML files.
## Tests
Covered by the tests in #2201
## Changes
The APIs at Databricks when returning a non `200` status code will
return a response body of the format:
```
{
"error_code": "Error code",
"message": "Human-readable error message."
}
```
This PR adds the ability to stub non-200 status codes in the test
server, allowing us to mock API errors from Databricks.
## Tests
New test
$VARNAME is what we use for environment variables, it's good to
separate.
Some people use envsubst for homemade variable interpolation, it's also
good to have separation there.
Bumps [astral-sh/ruff-action](https://github.com/astral-sh/ruff-action)
from 3.0.1 to 3.1.0.
<details>
<summary>Release notes</summary>
<p><em>Sourced from <a
href="https://github.com/astral-sh/ruff-action/releases">astral-sh/ruff-action's
releases</a>.</em></p>
<blockquote>
<h2>v3.1.0 🌈 Determine ruff version from optional or dependency
groups</h2>
<h2>Changes</h2>
<p>Big thank you to <a
href="https://github.com/AA-Turner"><code>@AA-Turner</code></a> for
expanding the pyproject.toml parsing to also find the ruff version to
use in the following scenarios:</p>
<pre lang="toml"><code>[dependency-groups]
dev = [
{ include-group = "docs" },
{ include-group = "lint" },
]
docs = [
"sphinx",
]
lint = [
"ruff==0.8.3",
]
</code></pre>
<pre lang="toml"><code>[project.optional-dependencies]
lint = [
"ruff==0.8.3",
]
</code></pre>
<h2>🚀 Enhancements</h2>
<ul>
<li>Read the <code>[project.optional-dependencies]</code> and
<code>[dependency-groups]</code> tables <a
href="https://github.com/AA-Turner"><code>@AA-Turner</code></a> (<a
href="https://redirect.github.com/astral-sh/ruff-action/issues/66">#66</a>)</li>
</ul>
<h2>v3.0.2 🌈 Full support for GHES</h2>
<h2>Changes</h2>
<p>This release fixes some issues that prevented use with GitHub
Enterprise Server instances.
Parsing the ruff version from pyproject.toml now also uses a library
that is fully TOML 1.0.0 compliant.</p>
<h2>🐛 Bug fixes</h2>
<ul>
<li>Do not expect GITHUB_TOKEN to be set or valid <a
href="https://github.com/eifinger"><code>@eifinger</code></a> (<a
href="https://redirect.github.com/astral-sh/ruff-action/issues/65">#65</a>)</li>
<li>Use TOML 1.0.0 compliant library for parsing <a
href="https://github.com/eifinger"><code>@eifinger</code></a> (<a
href="https://redirect.github.com/astral-sh/ruff-action/issues/47">#47</a>)</li>
</ul>
<h2>🧰 Maintenance</h2>
<ul>
<li>Fix compiled known versions <a
href="https://github.com/eifinger"><code>@eifinger</code></a> (<a
href="https://redirect.github.com/astral-sh/ruff-action/issues/62">#62</a>)</li>
<li>chore: update known checksums for 0.9.3 @<a
href="https://github.com/apps/github-actions">github-actions[bot]</a>
(<a
href="https://redirect.github.com/astral-sh/ruff-action/issues/61">#61</a>)</li>
<li>chore: update known checksums for 0.9.1 @<a
href="https://github.com/apps/github-actions">github-actions[bot]</a>
(<a
href="https://redirect.github.com/astral-sh/ruff-action/issues/42">#42</a>)</li>
</ul>
<h2>📚 Documentation</h2>
<ul>
<li>Fix Markdown link to Install the latest version <a
href="https://github.com/eifinger"><code>@eifinger</code></a> (<a
href="https://redirect.github.com/astral-sh/ruff-action/issues/58">#58</a>)</li>
</ul>
<!-- raw HTML omitted -->
</blockquote>
<p>... (truncated)</p>
</details>
<details>
<summary>Commits</summary>
<ul>
<li><a
href="f14634c415"><code>f14634c</code></a>
Read the <code>[project.optional-dependencies]</code> and
<code>[dependency-groups]</code> tables (...</li>
<li><a
href="47de3deae8"><code>47de3de</code></a>
Bump <code>@types/node</code> from 22.10.10 to 22.12.0 (<a
href="https://redirect.github.com/astral-sh/ruff-action/issues/60">#60</a>)</li>
<li><a
href="d8281c74d4"><code>d8281c7</code></a>
Do not expect GITHUB_TOKEN to be set or valid (<a
href="https://redirect.github.com/astral-sh/ruff-action/issues/65">#65</a>)</li>
<li><a
href="a634044659"><code>a634044</code></a>
Bump eifinger/actionlint-action from 1.9.0 to 1.9.1 (<a
href="https://redirect.github.com/astral-sh/ruff-action/issues/59">#59</a>)</li>
<li><a
href="2993ff4a65"><code>2993ff4</code></a>
Fix compiled known versions (<a
href="https://redirect.github.com/astral-sh/ruff-action/issues/62">#62</a>)</li>
<li><a
href="20a3b171f4"><code>20a3b17</code></a>
chore: update known checksums for 0.9.3 (<a
href="https://redirect.github.com/astral-sh/ruff-action/issues/61">#61</a>)</li>
<li><a
href="1c1aef9e3d"><code>1c1aef9</code></a>
Bump typescript from 5.7.2 to 5.7.3 (<a
href="https://redirect.github.com/astral-sh/ruff-action/issues/41">#41</a>)</li>
<li><a
href="0ceb04d9a0"><code>0ceb04d</code></a>
Bump release-drafter/release-drafter from 6.0.0 to 6.1.0 (<a
href="https://redirect.github.com/astral-sh/ruff-action/issues/50">#50</a>)</li>
<li><a
href="18db80c954"><code>18db80c</code></a>
Bump <code>@types/node</code> from 22.10.5 to 22.10.10 (<a
href="https://redirect.github.com/astral-sh/ruff-action/issues/53">#53</a>)</li>
<li><a
href="0a5dfb89f1"><code>0a5dfb8</code></a>
Fix Markdown link to Install the latest version (<a
href="https://redirect.github.com/astral-sh/ruff-action/issues/58">#58</a>)</li>
<li>Additional commits viewable in <a
href="31a5185046...f14634c415">compare
view</a></li>
</ul>
</details>
<br />
[](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores)
Dependabot will resolve any conflicts with this PR as long as you don't
alter it yourself. You can also trigger a rebase manually by commenting
`@dependabot rebase`.
[//]: # (dependabot-automerge-start)
[//]: # (dependabot-automerge-end)
---
<details>
<summary>Dependabot commands and options</summary>
<br />
You can trigger Dependabot actions by commenting on this PR:
- `@dependabot rebase` will rebase this PR
- `@dependabot recreate` will recreate this PR, overwriting any edits
that have been made to it
- `@dependabot merge` will merge this PR after your CI passes on it
- `@dependabot squash and merge` will squash and merge this PR after
your CI passes on it
- `@dependabot cancel merge` will cancel a previously requested merge
and block automerging
- `@dependabot reopen` will reopen this PR if it is closed
- `@dependabot close` will close this PR and stop Dependabot recreating
it. You can achieve the same result by closing it manually
- `@dependabot show <dependency name> ignore conditions` will show all
of the ignore conditions of the specified dependency
- `@dependabot ignore this major version` will close this PR and stop
Dependabot creating any more for this major version (unless you reopen
the PR or upgrade to it yourself)
- `@dependabot ignore this minor version` will close this PR and stop
Dependabot creating any more for this minor version (unless you reopen
the PR or upgrade to it yourself)
- `@dependabot ignore this dependency` will close this PR and stop
Dependabot creating any more for this dependency (unless you reopen the
PR or upgrade to it yourself)
</details>
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
## Changes
- Include acceptance directory in integration tests. Acceptance tests
will not start local server if CLOUD_ENV is set, so they become
integration tests.
- Add dependency for vendor to integration, so that CLI can be build
there.
- Implement LocalOnly option in test.toml to opt out of running
acceptance tests as integration tests. Use it in certain tests that are
difficult or not necessary to fix when run as integration tests.
- Update terraform test to redact timings out.
- Clean up .workspace.current_user from outputs of the tests.
## Tests
Existing tests.
## Changes
- Include compact relPath in the error message title. Include full paths
in separate lines below.
- Previously sometimes full paths were printed, sometime only rel path.
## Tests
Manually trigger the errors.
It is proven to be not necessary.
```
~/work/cli/acceptance % hyperfine -w 2 'go test' # with change:
Benchmark 1: go test
Time (mean ± σ): 4.983 s ± 0.209 s [User: 6.073 s, System: 9.869 s]
Range (min … max): 4.792 s … 5.483 s 10 runs
~/work/cli/acceptance % git stash # without change:
~/work/cli/acceptance % hyperfine -w 2 'go test'
Benchmark 1: go test
Time (mean ± σ): 5.018 s ± 0.100 s [User: 6.142 s, System: 10.234 s]
Range (min … max): 4.899 s … 5.182 s 10 runs
```
## Changes
- Add a script install_terraform.py that downloads terraform and
provider and generates a config to use, inspired by
https://gist.github.com/pietern/1cb6b6f3e0a452328e13cdc75031105e
- Make acceptance tests run this script once before running the tests
and set the required env vars to make cli use this terraform
installation.
- Use OS-specific directory for things that are build by acceptance test
runner (CLI and terraform).
This enables acceptance tests against cloud #2242 and local test for
bundle deploy #2254.
## Tests
- Add an acceptance test for standalone terraform. This is useful to
debug terraform with TF_LOG=DEBUG to see that it uses local provider.
- Other acceptance tests are updated with regard to terraform exec path.
- The overall time for tests locally is unchanged (if terraform is
already fetched).
## Changes
With this PR, any acceptance tests that define custom server stubs in
`test.toml` will automatically record all HTTP requests made and assert
on them.
Builds on top of https://github.com/databricks/cli/pull/2226
## Tests
Modifying existing acceptance test.
## Changes
This PR allows us to define custom server stubs in a `test.toml` file.
Note: A followup PR will add functionality to do assertions on the API
request itself.
## Tests
New acceptance test.
The default is 88 which reformats too much.
This has no effect on templates but affects Python script in this PR
https://github.com/databricks/cli/pull/2267
For context, we do not set any line length for golang and have 177 .go
files with max line length 150 or more.
## Changes
This PR registers the `server.Close()` function to be run during test
cleanup in the server initialization function. This ensures that all
test servers are closed as soon as the test they are scoped to finish.
Motivated by https://github.com/databricks/cli/pull/2255/files where a
regression was introduced where we did not close the test server.
## Tests
N/A
## Changes
Followup from
https://github.com/databricks/cli/pull/2209#pullrequestreview-2580308075.
This PR adds an integration test to validate that the API type bindings
work against the telemetry endpoint.
## Tests
N/A
---------
Co-authored-by: Pieter Noordhuis <pieter.noordhuis@databricks.com>
## Changes
Fix relative path errors in the Python mutator that was failing during
deployment since v0.239.1.
Before that:
```
% databricks bundle deploy
Deploying resources...
Updating deployment state...
Error: failed to compute relative path for job jobs_as_code_project_job: Rel: can't make resources/jobs_as_code_project_job.py relative to /Users/$USER/jobs_as_code_project
```
As a result, the bundle was deployed, but the deployment state wasn't
updated.
## Tests
Unit tests, adding acceptance tests in
https://github.com/databricks/cli/pull/2254
## Changes
Added support for double underscore variable references.
Previously we made this restriction stronger with no particular reason,
TF provider supports multiple underscores and thus DABs should do as
well.
Fixes#1753
## Tests
Added acceptance and integration tests
## Changes
These types correspond to the telemetry protobufs defined in universe.
## Tests
No tests are needed since this PR only adds the type bindings.
---------
Co-authored-by: Pieter Noordhuis <pieter.noordhuis@databricks.com>
## Changes
This change is required to enable tracking execution time telemetry for
bundle commands. In order to track execution time for the command
generally, we need to have the databricks auth configuration available
at this section of the code:
41bbd89257/cmd/root/root.go (L99)
In order to do this we can rely on the `configUsed` context key.
Most commands rely on the `root.MustWorkspaceClient` function which
automatically sets the client config in the `configUsed` context key.
Bundle commands, however, do not do so. They instead store their
workspace clients in the `&bundle.Bundle{}` object.
With this PR, the `configUsed` context key will be set for all `bundle`
commands. Functionally nothing changes.
## Tests
Existing tests. Also manually verified that either
`root.MustConfigureBundle` or `utils.ConfigureBundleWithVariables` is
called for all bundle commands (except `bundle init`) thus ensuring this
context key would be set for all bundle commands.
refs for the functions:
1. `root.MustConfigureBundle`:
41bbd89257/cmd/root/bundle.go (L88)
2. `utils.ConfigureBundleWithVariables`:
41bbd89257/cmd/bundle/utils/utils.go (L19)
---------
Co-authored-by: Pieter Noordhuis <pieter.noordhuis@databricks.com>
## Changes
This allows DABs to avoid waiting for the compute to start when app is
initially created as part of "bundle deploy" which significantly
improves deploy time.
Always set no_compute to true for apps
## Tests
Covered by `TestDeployBundleWithApp`, currently fails until TF provider
is upgraded to the version supporting `no_compute` option
Denis Bilenko
Andrew Nester
shreyas-goenka
Gleb Kanterov
Ilya Kuznetsov
Marcin Wojtyczka
Pieter Noordhuis
rikjansen-hu
Simon Poltier
dependabot[bot]