mirror of https://github.com/databricks/cli.git
1283 Commits
| Author | SHA1 | Message | Date |
|---|---|---|---|
Shreyas Goenka
|
489aba8d76
|
wip | |
|
Shreyas Goenka
|
5abb009c4f
|
wip | |
Andrew Nester
|
f0e2981596
|
Added JSON input validation for CLI commands (#1771)
## Changes Added JSON input validation for CLI commands. Now when invalid JSON passed as a payload to CLI commands, CLI performs input normalisation and detects if there are any mismatches such as incorrect types, unknown fields and etc. This diagnostic information is printed in standard error output and does not block command execution, so the change is backward compatible. Fixes #1769 #1764 #1625 #1560 ## Tests Added unit tests ``` andrew.nester@HFW9Y94129 ~ % databricks jobs create --json '{"seeti}' Error: error decoding JSON at (inline):1:2: unexpected EOF andrew.nester@HFW9Y94129 ~ % databricks jobs create --json '{"seeti": true}' Warning: unknown field: seeti in (inline):1:9 Error: Job settings must be specified. ``` --------- Co-authored-by: Pieter Noordhuis <pieter.noordhuis@databricks.com> |
|
Lennart Kats (databricks)
|
08a0d083c3
|
Ignore metastore permission error during template generation (#1819)
## Changes
This extends the `{{default_catalog}}` helper in templates to ignore any
`PERMISSION_DENIED` error. We're still reviewing when exactly this error
occurs, but if it does, it should not break templates. We should fall
back to assuming there's no default catalog (and no UC) instead.
## Testing
I have not been able to reproduce this issue, but there is a customer
report about "access denied to clusters that don't have unity catalog
enabled" being returned on a non-UC workspace. The error code in this PR
corresponds to that message.
## Next steps
We'll work together with the UC team to review if this error even makes
sense for this API. If that discussion leads to a behavior change in the
API we can update the CLI code again.
|
|
|
Andrew Nester
|
845d23ac21
|
Fixed typo in converting cluster permissions (#1826)
## Changes Fixed typo in converting cluster permissions |
|
Pieter Noordhuis
|
b9085de533
|
Remove unused `IS_OWNER` constant (#1823)
## Changes Leftover from #1386. ## Tests All tests pass (indicating it really wasn't used). |
|
|
Pieter Noordhuis
|
3270afaff4
|
Move utility functions dealing with IAM to libs/iamutil (#1820)
## Changes The two functions `GetShortUserName` and `IsServicePrincipal` are unrelated to auth or the purpose of the auth package. This change moves them into their own package and updates `IsServicePrincipal` to take an `*iam.User` argument instead of a string username. ## Tests Tests pass. |
|
|
Lennart Kats (databricks)
|
e885794722
|
Show actionable errors for collaborative deployment scenarios (#1386)
## Changes This adds diagnostics for collaborative (production) deployment scenarios, including: - Bob deploys a bundle that is normally deployed by Alice, but this fails because Bob can't write to `/Users/Alice/.bundle`. - Charlie deploys a bundle that is normally deployed by Alice, but this fails because he can't create a new pipeline where Alice would be the owner. - Alice deploys a bundle where she didn't list herself as one of the CAN_MANAGE users in permissions. That can work, but is probably a mistake. ## Tests Unit tests, manual testing. |
|
|
Andrew Nester
|
c92c67afc9
|
[Release] Release v0.230.0 (#1817)
Notable changes for Databricks Asset Bundles:
Workspace paths are automatically prefixed with `/Workspace`. In
addition, all usage of path strings such as
`/Workspace/${workspace.root_path}/...` in bundle configuration is
automatically replaced with `${workspace.root_path}/...` and generates a
warning as part of bundle validate.
More details can be find here:
https://docs.databricks.com/en/release-notes/dev-tools/bundles.html#workspace-paths-will-be-automatically-prefixed
Bundles:
* Add an error if state files grow bigger than the export limit
([#1795](https://github.com/databricks/cli/pull/1795)).
* Always prepend bundle remote paths with /Workspace
([#1724](https://github.com/databricks/cli/pull/1724)).
* Add resource path field to bundle workspace configuration
([#1800](https://github.com/databricks/cli/pull/1800)).
* Add validation for files with a `.(resource-name).yml` extension
([#1780](https://github.com/databricks/cli/pull/1780)).
Internal:
* Remove deprecated or readonly fields from the bundle schema
([#1809](https://github.com/databricks/cli/pull/1809)).
API Changes:
* Changed `databricks git-credentials create`, `databricks
git-credentials delete`, `databricks git-credentials get`, `databricks
git-credentials list`, `databricks git-credentials update` commands .
* Changed `databricks repos create`, `databricks repos delete`,
`databricks repos get`, `databricks repos update` command .
OpenAPI commit 0c86ea6dbd9a730c24ff0d4e509603e476955ac5 (2024-10-02)
Dependency updates:
* Upgrade TF provider to 1.53.0
([#1815](https://github.com/databricks/cli/pull/1815)).
* Bump golang.org/x/term from 0.24.0 to 0.25.0
([#1811](https://github.com/databricks/cli/pull/1811)).
* Bump golang.org/x/text from 0.18.0 to 0.19.0
([#1812](https://github.com/databricks/cli/pull/1812)).
* Bump github.com/databricks/databricks-sdk-go from 0.47.0 to 0.48.0
([#1810](https://github.com/databricks/cli/pull/1810)).
---------
Co-authored-by: Pieter Noordhuis <pieter.noordhuis@databricks.com>
|
|
|
Pieter Noordhuis
|
ae568743c5
|
Upgrade TF provider to 1.53.0 (#1815)
## Changes See https://github.com/databricks/terraform-provider-databricks/releases/tag/v1.53.0 ## Tests Integration tests pass. |
|
shreyas-goenka
|
88318d384a
|
Remove deprecated or readonly fields from the bundle schema (#1809)
## Changes We do not need the user to specify these fields in their bundle configuration, so we remove them from the JSON schema. ## Tests Manually and end to end tests. The JSON schema has also been regenerated after these changes. |
|
dependabot[bot]
|
7a2141fc5b
|
Bump github.com/databricks/databricks-sdk-go from 0.47.0 to 0.48.0 (#1810)
Bumps [github.com/databricks/databricks-sdk-go](https://github.com/databricks/databricks-sdk-go) from 0.47.0 to 0.48.0. <details> <summary>Release notes</summary> <p><em>Sourced from <a href="https://github.com/databricks/databricks-sdk-go/releases">github.com/databricks/databricks-sdk-go's releases</a>.</em></p> <blockquote> <h2>v0.48.0</h2> <h3>Internal Changes</h3> <ul> <li>Update SDK to latest OpenAPI spec (<a href="https://redirect.github.com/databricks/databricks-sdk-go/pull/1057">#1057</a>).</li> </ul> <p>Note: This release contains breaking changes, please see the API changes below for more details.</p> <h3>API Changes:</h3> <ul> <li>Added <code>DefaultSourceCodePath</code> and <code>Resources</code> fields for <a href="https://pkg.go.dev/github.com/databricks/databricks-sdk-go/service/apps#App">apps.App</a>.</li> <li>Added <code>Resources</code> field for <a href="https://pkg.go.dev/github.com/databricks/databricks-sdk-go/service/apps#CreateAppRequest">apps.CreateAppRequest</a>.</li> <li>Added <code>Resources</code> field for <a href="https://pkg.go.dev/github.com/databricks/databricks-sdk-go/service/apps#UpdateAppRequest">apps.UpdateAppRequest</a>.</li> <li>Added <code>Schema</code> field for <a href="https://pkg.go.dev/github.com/databricks/databricks-sdk-go/service/pipelines#CreatePipeline">pipelines.CreatePipeline</a>.</li> <li>Added <code>Schema</code> field for <a href="https://pkg.go.dev/github.com/databricks/databricks-sdk-go/service/pipelines#EditPipeline">pipelines.EditPipeline</a>.</li> <li>Added <code>Schema</code> field for <a href="https://pkg.go.dev/github.com/databricks/databricks-sdk-go/service/pipelines#PipelineSpec">pipelines.PipelineSpec</a>.</li> <li>[Breaking] Changed <code>Create</code> method for <a href="https://pkg.go.dev/github.com/databricks/databricks-sdk-go/service/workspace#GitCredentialsAPI">w.GitCredentials</a> workspace-level service . New request type is <a href="https://pkg.go.dev/github.com/databricks/databricks-sdk-go/service/workspace#CreateCredentialsRequest">workspace.CreateCredentialsRequest</a>.</li> <li>[Breaking] Changed <code>Create</code> method for <a href="https://pkg.go.dev/github.com/databricks/databricks-sdk-go/service/workspace#GitCredentialsAPI">w.GitCredentials</a> workspace-level service to type <code>Create</code> method for <a href="https://pkg.go.dev/github.com/databricks/databricks-sdk-go/service/workspace#GitCredentialsAPI">w.GitCredentials</a> workspace-level service.</li> <li>[Breaking] Changed <code>Delete</code> method for <a href="https://pkg.go.dev/github.com/databricks/databricks-sdk-go/service/workspace#GitCredentialsAPI">w.GitCredentials</a> workspace-level service . New request type is <a href="https://pkg.go.dev/github.com/databricks/databricks-sdk-go/service/workspace#DeleteCredentialsRequest">workspace.DeleteCredentialsRequest</a>.</li> <li>[Breaking] Changed <code>Delete</code> method for <a href="https://pkg.go.dev/github.com/databricks/databricks-sdk-go/service/workspace#GitCredentialsAPI">w.GitCredentials</a> workspace-level service to return <code>any</code>.</li> <li>Changed <code>Delete</code> method for <a href="https://pkg.go.dev/github.com/databricks/databricks-sdk-go/service/workspace#GitCredentialsAPI">w.GitCredentials</a> workspace-level service to type <code>Delete</code> method for <a href="https://pkg.go.dev/github.com/databricks/databricks-sdk-go/service/workspace#GitCredentialsAPI">w.GitCredentials</a> workspace-level service.</li> <li>[Breaking] Changed <code>Get</code> method for <a href="https://pkg.go.dev/github.com/databricks/databricks-sdk-go/service/workspace#GitCredentialsAPI">w.GitCredentials</a> workspace-level service . New request type is <a href="https://pkg.go.dev/github.com/databricks/databricks-sdk-go/service/workspace#GetCredentialsRequest">workspace.GetCredentialsRequest</a>.</li> <li>Changed <code>Get</code> method for <a href="https://pkg.go.dev/github.com/databricks/databricks-sdk-go/service/workspace#GitCredentialsAPI">w.GitCredentials</a> workspace-level service to type <code>Get</code> method for <a href="https://pkg.go.dev/github.com/databricks/databricks-sdk-go/service/workspace#GitCredentialsAPI">w.GitCredentials</a> workspace-level service.</li> <li>[Breaking] Changed <code>Get</code> method for <a href="https://pkg.go.dev/github.com/databricks/databricks-sdk-go/service/workspace#GitCredentialsAPI">w.GitCredentials</a> workspace-level service to return <a href="https://pkg.go.dev/github.com/databricks/databricks-sdk-go/service/workspace#GetCredentialsResponse">workspace.GetCredentialsResponse</a>.</li> <li>[Breaking] Changed <code>List</code> method for <a href="https://pkg.go.dev/github.com/databricks/databricks-sdk-go/service/workspace#GitCredentialsAPI">w.GitCredentials</a> workspace-level service to return <a href="https://pkg.go.dev/github.com/databricks/databricks-sdk-go/service/workspace#ListCredentialsResponse">workspace.ListCredentialsResponse</a>.</li> <li>Changed <code>List</code> method for <a href="https://pkg.go.dev/github.com/databricks/databricks-sdk-go/service/workspace#GitCredentialsAPI">w.GitCredentials</a> workspace-level service to type <code>List</code> method for <a href="https://pkg.go.dev/github.com/databricks/databricks-sdk-go/service/workspace#GitCredentialsAPI">w.GitCredentials</a> workspace-level service.</li> <li>Changed <code>Update</code> method for <a href="https://pkg.go.dev/github.com/databricks/databricks-sdk-go/service/workspace#GitCredentialsAPI">w.GitCredentials</a> workspace-level service to type <code>Update</code> method for <a href="https://pkg.go.dev/github.com/databricks/databricks-sdk-go/service/workspace#GitCredentialsAPI">w.GitCredentials</a> workspace-level service.</li> <li>[Breaking] Changed <code>Update</code> method for <a href="https://pkg.go.dev/github.com/databricks/databricks-sdk-go/service/workspace#GitCredentialsAPI">w.GitCredentials</a> workspace-level service to return <code>any</code>.</li> <li>[Breaking] Changed <code>Update</code> method for <a href="https://pkg.go.dev/github.com/databricks/databricks-sdk-go/service/workspace#GitCredentialsAPI">w.GitCredentials</a> workspace-level service . New request type is <a href="https://pkg.go.dev/github.com/databricks/databricks-sdk-go/service/workspace#UpdateCredentialsRequest">workspace.UpdateCredentialsRequest</a>.</li> <li>Changed <code>Create</code> method for <a href="https://pkg.go.dev/github.com/databricks/databricks-sdk-go/service/workspace#ReposAPI">w.Repos</a> workspace-level service to type <code>Create</code> method for <a href="https://pkg.go.dev/github.com/databricks/databricks-sdk-go/service/workspace#ReposAPI">w.Repos</a> workspace-level service.</li> <li>[Breaking] Changed <code>Create</code> method for <a href="https://pkg.go.dev/github.com/databricks/databricks-sdk-go/service/workspace#ReposAPI">w.Repos</a> workspace-level service . New request type is <a href="https://pkg.go.dev/github.com/databricks/databricks-sdk-go/service/workspace#CreateRepoRequest">workspace.CreateRepoRequest</a>.</li> <li>[Breaking] Changed <code>Create</code> method for <a href="https://pkg.go.dev/github.com/databricks/databricks-sdk-go/service/workspace#ReposAPI">w.Repos</a> workspace-level service to return <a href="https://pkg.go.dev/github.com/databricks/databricks-sdk-go/service/workspace#CreateRepoResponse">workspace.CreateRepoResponse</a>.</li> <li>[Breaking] Changed <code>Delete</code> method for <a href="https://pkg.go.dev/github.com/databricks/databricks-sdk-go/service/workspace#ReposAPI">w.Repos</a> workspace-level service to return <code>any</code>.</li> <li>Changed <code>Delete</code> method for <a href="https://pkg.go.dev/github.com/databricks/databricks-sdk-go/service/workspace#ReposAPI">w.Repos</a> workspace-level service to type <code>Delete</code> method for <a href="https://pkg.go.dev/github.com/databricks/databricks-sdk-go/service/workspace#ReposAPI">w.Repos</a> workspace-level service.</li> <li>[Breaking] Changed <code>Get</code> method for <a href="https://pkg.go.dev/github.com/databricks/databricks-sdk-go/service/workspace#ReposAPI">w.Repos</a> workspace-level service to return <a href="https://pkg.go.dev/github.com/databricks/databricks-sdk-go/service/workspace#GetRepoResponse">workspace.GetRepoResponse</a>.</li> <li>Changed <code>Get</code> method for <a href="https://pkg.go.dev/github.com/databricks/databricks-sdk-go/service/workspace#ReposAPI">w.Repos</a> workspace-level service to type <code>Get</code> method for <a href="https://pkg.go.dev/github.com/databricks/databricks-sdk-go/service/workspace#ReposAPI">w.Repos</a> workspace-level service.</li> <li>Changed <code>Update</code> method for <a href="https://pkg.go.dev/github.com/databricks/databricks-sdk-go/service/workspace#ReposAPI">w.Repos</a> workspace-level service to return <code>any</code>.</li> <li>[Breaking] Changed <code>Update</code> method for <a href="https://pkg.go.dev/github.com/databricks/databricks-sdk-go/service/workspace#ReposAPI">w.Repos</a> workspace-level service . New request type is <a href="https://pkg.go.dev/github.com/databricks/databricks-sdk-go/service/workspace#UpdateRepoRequest">workspace.UpdateRepoRequest</a>.</li> <li>Changed <code>Update</code> method for <a href="https://pkg.go.dev/github.com/databricks/databricks-sdk-go/service/workspace#ReposAPI">w.Repos</a> workspace-level service to type <code>Update</code> method for <a href="https://pkg.go.dev/github.com/databricks/databricks-sdk-go/service/workspace#ReposAPI">w.Repos</a> workspace-level service.</li> <li>[Breaking] Changed <code>CredentialId</code> and <code>GitProvider</code> fields for <a href="https://pkg.go.dev/github.com/databricks/databricks-sdk-go/service/workspace#CreateCredentialsResponse">workspace.CreateCredentialsResponse</a> to be required.</li> <li>Changed <code>CredentialId</code> field for <a href="https://pkg.go.dev/github.com/databricks/databricks-sdk-go/service/workspace#CredentialInfo">workspace.CredentialInfo</a> to be required.</li> <li>Changed <code>CredentialId</code> field for <a href="https://pkg.go.dev/github.com/databricks/databricks-sdk-go/service/workspace#GetCredentialsResponse">workspace.GetCredentialsResponse</a> to be required.</li> <li>Changed <code>Patterns</code> field for <a href="https://pkg.go.dev/github.com/databricks/databricks-sdk-go/service/workspace#SparseCheckout">workspace.SparseCheckout</a> to type <a href="https://pkg.go.dev/github.com/databricks/databricks-sdk-go/service/workspace#List">workspace.List</a>.</li> <li>Changed <code>Patterns</code> field for <a href="https://pkg.go.dev/github.com/databricks/databricks-sdk-go/service/workspace#SparseCheckoutUpdate">workspace.SparseCheckoutUpdate</a> to type <a href="https://pkg.go.dev/github.com/databricks/databricks-sdk-go/service/workspace#List">workspace.List</a>.</li> <li>[Breaking] Changed <code>GitProvider</code> field for <a href="https://pkg.go.dev/github.com/databricks/databricks-sdk-go/service/workspace#UpdateCredentialsRequest">workspace.UpdateCredentialsRequest</a> to be required.</li> </ul> <p>OpenAPI SHA: 0c86ea6dbd9a730c24ff0d4e509603e476955ac5, Date: 2024-10-02</p> </blockquote> </details> <details> <summary>Changelog</summary> <p><em>Sourced from <a href="https://github.com/databricks/databricks-sdk-go/blob/main/CHANGELOG.md">github.com/databricks/databricks-sdk-go's changelog</a>.</em></p> <blockquote> <h2>[Release] Release v0.48.0</h2> <h3>Internal Changes</h3> <ul> <li>Update SDK to latest OpenAPI spec (<a href="https://redirect.github.com/databricks/databricks-sdk-go/pull/1057">#1057</a>).</li> </ul> <p>Note: This release contains breaking changes, please see the API changes below for more details.</p> <h3>API Changes:</h3> <ul> <li>Added <code>DefaultSourceCodePath</code> and <code>Resources</code> fields for <a href="https://pkg.go.dev/github.com/databricks/databricks-sdk-go/service/apps#App">apps.App</a>.</li> <li>Added <code>Resources</code> field for <a href="https://pkg.go.dev/github.com/databricks/databricks-sdk-go/service/apps#CreateAppRequest">apps.CreateAppRequest</a>.</li> <li>Added <code>Resources</code> field for <a href="https://pkg.go.dev/github.com/databricks/databricks-sdk-go/service/apps#UpdateAppRequest">apps.UpdateAppRequest</a>.</li> <li>Added <code>Schema</code> field for <a href="https://pkg.go.dev/github.com/databricks/databricks-sdk-go/service/pipelines#CreatePipeline">pipelines.CreatePipeline</a>.</li> <li>Added <code>Schema</code> field for <a href="https://pkg.go.dev/github.com/databricks/databricks-sdk-go/service/pipelines#EditPipeline">pipelines.EditPipeline</a>.</li> <li>Added <code>Schema</code> field for <a href="https://pkg.go.dev/github.com/databricks/databricks-sdk-go/service/pipelines#PipelineSpec">pipelines.PipelineSpec</a>.</li> <li>[Breaking] Changed <code>Create</code> method for <a href="https://pkg.go.dev/github.com/databricks/databricks-sdk-go/service/workspace#GitCredentialsAPI">w.GitCredentials</a> workspace-level service . New request type is <a href="https://pkg.go.dev/github.com/databricks/databricks-sdk-go/service/workspace#CreateCredentialsRequest">workspace.CreateCredentialsRequest</a>.</li> <li>[Breaking] Changed <code>Create</code> method for <a href="https://pkg.go.dev/github.com/databricks/databricks-sdk-go/service/workspace#GitCredentialsAPI">w.GitCredentials</a> workspace-level service to type <code>Create</code> method for <a href="https://pkg.go.dev/github.com/databricks/databricks-sdk-go/service/workspace#GitCredentialsAPI">w.GitCredentials</a> workspace-level service.</li> <li>[Breaking] Changed <code>Delete</code> method for <a href="https://pkg.go.dev/github.com/databricks/databricks-sdk-go/service/workspace#GitCredentialsAPI">w.GitCredentials</a> workspace-level service . New request type is <a href="https://pkg.go.dev/github.com/databricks/databricks-sdk-go/service/workspace#DeleteCredentialsRequest">workspace.DeleteCredentialsRequest</a>.</li> <li>[Breaking] Changed <code>Delete</code> method for <a href="https://pkg.go.dev/github.com/databricks/databricks-sdk-go/service/workspace#GitCredentialsAPI">w.GitCredentials</a> workspace-level service to return <code>any</code>.</li> <li>Changed <code>Delete</code> method for <a href="https://pkg.go.dev/github.com/databricks/databricks-sdk-go/service/workspace#GitCredentialsAPI">w.GitCredentials</a> workspace-level service to type <code>Delete</code> method for <a href="https://pkg.go.dev/github.com/databricks/databricks-sdk-go/service/workspace#GitCredentialsAPI">w.GitCredentials</a> workspace-level service.</li> <li>[Breaking] Changed <code>Get</code> method for <a href="https://pkg.go.dev/github.com/databricks/databricks-sdk-go/service/workspace#GitCredentialsAPI">w.GitCredentials</a> workspace-level service . New request type is <a href="https://pkg.go.dev/github.com/databricks/databricks-sdk-go/service/workspace#GetCredentialsRequest">workspace.GetCredentialsRequest</a>.</li> <li>Changed <code>Get</code> method for <a href="https://pkg.go.dev/github.com/databricks/databricks-sdk-go/service/workspace#GitCredentialsAPI">w.GitCredentials</a> workspace-level service to type <code>Get</code> method for <a href="https://pkg.go.dev/github.com/databricks/databricks-sdk-go/service/workspace#GitCredentialsAPI">w.GitCredentials</a> workspace-level service.</li> <li>[Breaking] Changed <code>Get</code> method for <a href="https://pkg.go.dev/github.com/databricks/databricks-sdk-go/service/workspace#GitCredentialsAPI">w.GitCredentials</a> workspace-level service to return <a href="https://pkg.go.dev/github.com/databricks/databricks-sdk-go/service/workspace#GetCredentialsResponse">workspace.GetCredentialsResponse</a>.</li> <li>[Breaking] Changed <code>List</code> method for <a href="https://pkg.go.dev/github.com/databricks/databricks-sdk-go/service/workspace#GitCredentialsAPI">w.GitCredentials</a> workspace-level service to return <a href="https://pkg.go.dev/github.com/databricks/databricks-sdk-go/service/workspace#ListCredentialsResponse">workspace.ListCredentialsResponse</a>.</li> <li>Changed <code>List</code> method for <a href="https://pkg.go.dev/github.com/databricks/databricks-sdk-go/service/workspace#GitCredentialsAPI">w.GitCredentials</a> workspace-level service to type <code>List</code> method for <a href="https://pkg.go.dev/github.com/databricks/databricks-sdk-go/service/workspace#GitCredentialsAPI">w.GitCredentials</a> workspace-level service.</li> <li>Changed <code>Update</code> method for <a href="https://pkg.go.dev/github.com/databricks/databricks-sdk-go/service/workspace#GitCredentialsAPI">w.GitCredentials</a> workspace-level service to type <code>Update</code> method for <a href="https://pkg.go.dev/github.com/databricks/databricks-sdk-go/service/workspace#GitCredentialsAPI">w.GitCredentials</a> workspace-level service.</li> <li>[Breaking] Changed <code>Update</code> method for <a href="https://pkg.go.dev/github.com/databricks/databricks-sdk-go/service/workspace#GitCredentialsAPI">w.GitCredentials</a> workspace-level service to return <code>any</code>.</li> <li>[Breaking] Changed <code>Update</code> method for <a href="https://pkg.go.dev/github.com/databricks/databricks-sdk-go/service/workspace#GitCredentialsAPI">w.GitCredentials</a> workspace-level service . New request type is <a href="https://pkg.go.dev/github.com/databricks/databricks-sdk-go/service/workspace#UpdateCredentialsRequest">workspace.UpdateCredentialsRequest</a>.</li> <li>Changed <code>Create</code> method for <a href="https://pkg.go.dev/github.com/databricks/databricks-sdk-go/service/workspace#ReposAPI">w.Repos</a> workspace-level service to type <code>Create</code> method for <a href="https://pkg.go.dev/github.com/databricks/databricks-sdk-go/service/workspace#ReposAPI">w.Repos</a> workspace-level service.</li> <li>[Breaking] Changed <code>Create</code> method for <a href="https://pkg.go.dev/github.com/databricks/databricks-sdk-go/service/workspace#ReposAPI">w.Repos</a> workspace-level service . New request type is <a href="https://pkg.go.dev/github.com/databricks/databricks-sdk-go/service/workspace#CreateRepoRequest">workspace.CreateRepoRequest</a>.</li> <li>[Breaking] Changed <code>Create</code> method for <a href="https://pkg.go.dev/github.com/databricks/databricks-sdk-go/service/workspace#ReposAPI">w.Repos</a> workspace-level service to return <a href="https://pkg.go.dev/github.com/databricks/databricks-sdk-go/service/workspace#CreateRepoResponse">workspace.CreateRepoResponse</a>.</li> <li>[Breaking] Changed <code>Delete</code> method for <a href="https://pkg.go.dev/github.com/databricks/databricks-sdk-go/service/workspace#ReposAPI">w.Repos</a> workspace-level service to return <code>any</code>.</li> <li>Changed <code>Delete</code> method for <a href="https://pkg.go.dev/github.com/databricks/databricks-sdk-go/service/workspace#ReposAPI">w.Repos</a> workspace-level service to type <code>Delete</code> method for <a href="https://pkg.go.dev/github.com/databricks/databricks-sdk-go/service/workspace#ReposAPI">w.Repos</a> workspace-level service.</li> <li>[Breaking] Changed <code>Get</code> method for <a href="https://pkg.go.dev/github.com/databricks/databricks-sdk-go/service/workspace#ReposAPI">w.Repos</a> workspace-level service to return <a href="https://pkg.go.dev/github.com/databricks/databricks-sdk-go/service/workspace#GetRepoResponse">workspace.GetRepoResponse</a>.</li> <li>Changed <code>Get</code> method for <a href="https://pkg.go.dev/github.com/databricks/databricks-sdk-go/service/workspace#ReposAPI">w.Repos</a> workspace-level service to type <code>Get</code> method for <a href="https://pkg.go.dev/github.com/databricks/databricks-sdk-go/service/workspace#ReposAPI">w.Repos</a> workspace-level service.</li> <li>Changed <code>Update</code> method for <a href="https://pkg.go.dev/github.com/databricks/databricks-sdk-go/service/workspace#ReposAPI">w.Repos</a> workspace-level service to return <code>any</code>.</li> <li>[Breaking] Changed <code>Update</code> method for <a href="https://pkg.go.dev/github.com/databricks/databricks-sdk-go/service/workspace#ReposAPI">w.Repos</a> workspace-level service . New request type is <a href="https://pkg.go.dev/github.com/databricks/databricks-sdk-go/service/workspace#UpdateRepoRequest">workspace.UpdateRepoRequest</a>.</li> <li>Changed <code>Update</code> method for <a href="https://pkg.go.dev/github.com/databricks/databricks-sdk-go/service/workspace#ReposAPI">w.Repos</a> workspace-level service to type <code>Update</code> method for <a href="https://pkg.go.dev/github.com/databricks/databricks-sdk-go/service/workspace#ReposAPI">w.Repos</a> workspace-level service.</li> <li>[Breaking] Changed <code>CredentialId</code> and <code>GitProvider</code> fields for <a href="https://pkg.go.dev/github.com/databricks/databricks-sdk-go/service/workspace#CreateCredentialsResponse">workspace.CreateCredentialsResponse</a> to be required.</li> <li>Changed <code>CredentialId</code> field for <a href="https://pkg.go.dev/github.com/databricks/databricks-sdk-go/service/workspace#CredentialInfo">workspace.CredentialInfo</a> to be required.</li> <li>Changed <code>CredentialId</code> field for <a href="https://pkg.go.dev/github.com/databricks/databricks-sdk-go/service/workspace#GetCredentialsResponse">workspace.GetCredentialsResponse</a> to be required.</li> <li>Changed <code>Patterns</code> field for <a href="https://pkg.go.dev/github.com/databricks/databricks-sdk-go/service/workspace#SparseCheckout">workspace.SparseCheckout</a> to type <a href="https://pkg.go.dev/github.com/databricks/databricks-sdk-go/service/workspace#List">workspace.List</a>.</li> <li>Changed <code>Patterns</code> field for <a href="https://pkg.go.dev/github.com/databricks/databricks-sdk-go/service/workspace#SparseCheckoutUpdate">workspace.SparseCheckoutUpdate</a> to type <a href="https://pkg.go.dev/github.com/databricks/databricks-sdk-go/service/workspace#List">workspace.List</a>.</li> <li>[Breaking] Changed <code>GitProvider</code> field for <a href="https://pkg.go.dev/github.com/databricks/databricks-sdk-go/service/workspace#UpdateCredentialsRequest">workspace.UpdateCredentialsRequest</a> to be required.</li> </ul> <p>OpenAPI SHA: 0c86ea6dbd9a730c24ff0d4e509603e476955ac5, Date: 2024-10-02</p> </blockquote> </details> <details> <summary>Commits</summary> <ul> <li><a href=" |
|
|
dependabot[bot]
|
02bfd397ef
|
Bump golang.org/x/text from 0.18.0 to 0.19.0 (#1812)
Bumps [golang.org/x/text](https://github.com/golang/text) from 0.18.0 to 0.19.0. <details> <summary>Commits</summary> <ul> <li><a href=" |
|
|
dependabot[bot]
|
589e5ff1a4
|
Bump golang.org/x/term from 0.24.0 to 0.25.0 (#1811)
Bumps [golang.org/x/term](https://github.com/golang/term) from 0.24.0 to
0.25.0.
<details>
<summary>Commits</summary>
<ul>
<li><a
href="
|
|
|
shreyas-goenka
|
bca9c2eda4
|
Add validation for files with a `.(resource-name).yml` extension (#1780)
## Changes
We want to encourage a pattern of specifying only a single resource in a
YAML file when the `.(resource-type).yml` extension is used (for
example, `.job.yml`). This convention could allow us to bijectively map
a resource YAML file to its corresponding resource in the Databricks
workspace.
This PR:
1. Emits a recommendation diagnostic when we detect this convention is
being violated. We can promote this to a warning when we want to
encourage this pattern more strongly.
2. Visualises the recommendation diagnostics in the `bundle validate`
command.
**NOTE:** While this PR also shows the recommendation for `.yaml` files,
we do not encourage users to use this extension. We only support it here
since it's part of the YAML standard and some existing users might
already be using `.yaml`.
## Tests
Unit tests and manually. Here's what an example output looks like:
```
Recommendation: define a single job in a file with the .job.yml extension.
at resources.jobs.bar
resources.jobs.foo
in foo.job.yml:13:7
foo.job.yml:5:7
The following resources are defined or configured in this file:
- bar (job)
- foo (job)
```
---------
Co-authored-by: Lennart Kats (databricks) <lennart.kats@databricks.com>
|
|
|
Andrew Nester
|
a8cff48c0b
|
Always prepend bundle remote paths with /Workspace (#1724)
## Changes
Due to platform changes, all libraries, notebooks and etc. paths used in
Databricks must be started with either /Workspace or /Volumes prefix.
This PR makes sure that all bundle paths are correctly prefixed.
Note: this change is a breaking change if user previously configured and
used `/Workspace/Workspace` folder in their workspace file system or
having `/Workspace/${workspace.root_path}...` pattern configured
anywhere in their bundle config
Fixes: #1751
AI:
- [x] Scan DABs config and error out on
`/Workspace/${workspace.root_path}...` pattern usage
## Tests
Added unit tests
---------
Co-authored-by: Pieter Noordhuis <pieter.noordhuis@databricks.com>
|
|
|
Pieter Noordhuis
|
80d55f4540
|
Add resource path field to bundle workspace configuration (#1800)
## Changes Default workspace path for resources with a presence in the workspace tree. Note: this path is **not** created automatically (yet). We need this only for dashboards (so far), so can take care of creation if one or more dashboards are part of a deployment. This saves an API call for deployments where this is not necessary. ## Tests Expanded existing tests. |
|
|
Andrew Nester
|
044a00c7f9
|
Add an error if state files grow bigger than the export limit (#1795)
## Changes Currently API limits on exporting files from workspaces are set at 10 MBs while uploading to is 500 MBs. We want to prevent users running into deadlock when they won't be able to pull state file anymore so we prevent from uploading large state files (over 10 MBs) to Databricks workspace. |
|
|
Andrew Nester
|
c28d64f2dc
|
[Release] Release v0.229.0 (#1801)
Bundles: * Added support for creating all-purpose clusters ([#1698](https://github.com/databricks/cli/pull/1698)). * Reduce time until the prompt is shown for bundle run ([#1727](https://github.com/databricks/cli/pull/1727)). * Use Unity Catalog for pipelines in the default-python template ([#1766](https://github.com/databricks/cli/pull/1766)). * Add verbose flag to the "bundle deploy" command ([#1774](https://github.com/databricks/cli/pull/1774)). * Fixed full variable override detection ([#1787](https://github.com/databricks/cli/pull/1787)). * Add sub-extension to resource files in built-in templates ([#1777](https://github.com/databricks/cli/pull/1777)). * Fix panic in `apply_presets.go` ([#1796](https://github.com/databricks/cli/pull/1796)). Internal: * Assert tokens are redacted in origin URL when username is not specified ([#1785](https://github.com/databricks/cli/pull/1785)). * Refactor jobs path translation ([#1782](https://github.com/databricks/cli/pull/1782)). * Add JobTaskClusterSpec validate mutator ([#1784](https://github.com/databricks/cli/pull/1784)). * Pin Go toolchain to 1.22.7 ([#1790](https://github.com/databricks/cli/pull/1790)). * Modify SetLocation test utility to take full locations as argument ([#1788](https://github.com/databricks/cli/pull/1788)). * Simplified isFullVariableOverrideDef implementation ([#1791](https://github.com/databricks/cli/pull/1791)). * Sort tasks by `task_key` before generating the Terraform configuration ([#1776](https://github.com/databricks/cli/pull/1776)). * Trim trailing whitespace ([#1794](https://github.com/databricks/cli/pull/1794)). * Move trampoline code into trampoline package ([#1793](https://github.com/databricks/cli/pull/1793)). * Rename `RootPath` -> `BundleRootPath` ([#1792](https://github.com/databricks/cli/pull/1792)). API Changes: * Changed `databricks apps delete` command to return . * Changed `databricks apps deploy` command with new required argument order. * Changed `databricks apps start` command to return . * Changed `databricks apps stop` command to return . * Added `databricks temporary-table-credentials` command group. * Added `databricks serving-endpoints put-ai-gateway` command. * Added `databricks disable-legacy-access` command group. * Added `databricks account disable-legacy-features` command group. OpenAPI commit 6f6b1371e640f2dfeba72d365ac566368656f6b6 (2024-09-19) Dependency updates: * Upgrade to Go SDK 0.47.0 ([#1799](https://github.com/databricks/cli/pull/1799)). * Upgrade to TF provider 1.52 ([#1781](https://github.com/databricks/cli/pull/1781)). * Bump golang.org/x/mod from 0.20.0 to 0.21.0 ([#1758](https://github.com/databricks/cli/pull/1758)). * Bump github.com/hashicorp/hc-install from 0.7.0 to 0.9.0 ([#1772](https://github.com/databricks/cli/pull/1772)). |
|
|
Andrew Nester
|
84fc1ed131
|
Upgrade to Go SDK 0.47.0 (#1799)
## Changes Upgrade to Go SDK 0.47.0 ## Tests <!-- How is this tested? --> |
|
|
Lennart Kats (databricks)
|
da3b4f7c72
|
Fix panic in `apply_presets.go` (#1796)
## Changes This fixes the user-reported panic in `apply_presets.go`. I'm still unsure how to reproduce this, since the CLI just reports `ob broken_job is not defined` when I try to use `bundle deploy` with an empty job. That said — we may as well be defensive here and I see we have lots of checks for empty job/cluster/etc. settings scattered throughout our code base so at least we're somewhat consistent. |
|
|
Pieter Noordhuis
|
1d1aa0a416
|
Rename `RootPath` -> `BundleRootPath` (#1792)
## Changes After introducing the `SyncRootPath` field on the bundle (#1694), the previous `RootPath` became ambiguous. Does it mean the bundle root path or the sync root path? This PR renames to field to `BundleRootPath` to remove the ambiguity. ## Tests n/a --------- Co-authored-by: shreyas-goenka <88374338+shreyas-goenka@users.noreply.github.com> |
|
|
Pieter Noordhuis
|
56cd96cb93
|
Move trampoline code into trampoline package (#1793)
## Changes Doing this to make room for PyDABs under `bundle/python`. ## Tests n/a |
|
|
Pieter Noordhuis
|
a1dca56abf
|
Trim trailing whitespace (#1794)
## Changes Trailing whitespace is trimmed per the VS Code settings for this repository. ## Tests n/a |
|
|
shreyas-goenka
|
4e8e027380
|
Sort tasks by `task_key` before generating the Terraform configuration (#1776)
## Changes Sort the tasks in the resultant `bundle.tf.json`. This is important because configuration from one task can leak into another if the tasks are not sorted. For more details see: 1. https://github.com/databricks/terraform-provider-databricks/issues/3951 2. https://github.com/databricks/terraform-provider-databricks/issues/4011 ## Tests Unit test and manually. For manual testing I used the following configuration: ``` resources: jobs: foo: tasks: - task_key: task-Z notebook_task: notebook_path: nb.py source: GIT existing_cluster_id: 0715-133738-ju0ma84z - task_key: task-1 notebook_task: notebook_path: ${workspace.file_path}/local.py source: WORKSPACE existing_cluster_id: 0715-133738-ju0ma84z depends_on: - task_key: task-Z git_source: git_provider: gitHub git_url: https://github.com/shreyas-goenka/job-source-tmp.git git_branch: main ``` Steps (1): 1. Deploy this bundle. 2. Comment out "source: GIT" 3. Deploy again Before: Deploying this bundle twice would fail. This is because the "source: GIT" would carry over to the next deployment. After: There was no error on the subsequent deployment. Steps (2): 1. Deploy once 2. Deploy again Before: Works correctly but leads to a update API call every time. After: No diff is detected by terraform. |
|
|
Andrew Nester
|
66f2ba64a8
|
Simplified isFullVariableOverrideDef implementation (#1791)
## Changes Simplified isFullVariableOverrideDef implementation Follow up on https://github.com/databricks/cli/pull/1787 |
|
|
dependabot[bot]
|
94d8c3ba1e
|
Bump github.com/hashicorp/hc-install from 0.7.0 to 0.9.0 (#1772)
Bumps [github.com/hashicorp/hc-install](https://github.com/hashicorp/hc-install) from 0.7.0 to 0.9.0. <details> <summary>Release notes</summary> <p><em>Sourced from <a href="https://github.com/hashicorp/hc-install/releases">github.com/hashicorp/hc-install's releases</a>.</em></p> <blockquote> <h2>v0.9.0</h2> <h2>What's Changed</h2> <ul> <li>Finish Release of 0.8.1 by updating VERSION by <a href="https://github.com/mutahhir"><code>@mutahhir</code></a> in <a href="https://redirect.github.com/hashicorp/hc-install/pull/248">hashicorp/hc-install#248</a></li> <li>build(deps): bump golang.org/x/mod from 0.20.0 to 0.21.0 by <a href="https://github.com/dependabot"><code>@dependabot</code></a> in <a href="https://redirect.github.com/hashicorp/hc-install/pull/242">hashicorp/hc-install#242</a></li> <li>docs: Update release instructions by <a href="https://github.com/radeksimko"><code>@radeksimko</code></a> in <a href="https://redirect.github.com/hashicorp/hc-install/pull/249">hashicorp/hc-install#249</a></li> <li>Prepare for v0.9.0 release by <a href="https://github.com/mutahhir"><code>@mutahhir</code></a> in <a href="https://redirect.github.com/hashicorp/hc-install/pull/250">hashicorp/hc-install#250</a></li> </ul> <p><strong>Full Changelog</strong>: <a href="https://github.com/hashicorp/hc-install/compare/v0.8.1...v0.9.0">https://github.com/hashicorp/hc-install/compare/v0.8.1...v0.9.0</a></p> <h2>v0.8.1</h2> <h2>What's Changed</h2> <ul> <li>Add artifacts manifest (automatically generated) by <a href="https://github.com/jeanneryan"><code>@jeanneryan</code></a> in <a href="https://redirect.github.com/hashicorp/hc-install/pull/235">hashicorp/hc-install#235</a></li> <li>build(deps): bump golang.org/x/mod from 0.19.0 to 0.20.0 by <a href="https://github.com/dependabot"><code>@dependabot</code></a> in <a href="https://redirect.github.com/hashicorp/hc-install/pull/236">hashicorp/hc-install#236</a></li> <li>build(deps): Bump workflows to latest trusted versions by <a href="https://github.com/hashicorp-tsccr"><code>@hashicorp-tsccr</code></a> in <a href="https://redirect.github.com/hashicorp/hc-install/pull/237">hashicorp/hc-install#237</a></li> <li>build(deps): Bump workflows to latest trusted versions by <a href="https://github.com/hashicorp-tsccr"><code>@hashicorp-tsccr</code></a> in <a href="https://redirect.github.com/hashicorp/hc-install/pull/238">hashicorp/hc-install#238</a></li> <li>build(deps): bump hashicorp/action-setup-bob from 2.1.0 to 2.1.1 in the github-actions-backward-compatible group by <a href="https://github.com/dependabot"><code>@dependabot</code></a> in <a href="https://redirect.github.com/hashicorp/hc-install/pull/241">hashicorp/hc-install#241</a></li> <li>httpclient: Reuse existing configured logger by <a href="https://github.com/radeksimko"><code>@radeksimko</code></a> in <a href="https://redirect.github.com/hashicorp/hc-install/pull/240">hashicorp/hc-install#240</a></li> <li>build(deps): Bump workflows to latest trusted versions by <a href="https://github.com/hashicorp-tsccr"><code>@hashicorp-tsccr</code></a> in <a href="https://redirect.github.com/hashicorp/hc-install/pull/243">hashicorp/hc-install#243</a></li> <li>Nightly and PR builds: fix "no space left on device" on macOS runner by <a href="https://github.com/kmoe"><code>@kmoe</code></a> in <a href="https://redirect.github.com/hashicorp/hc-install/pull/245">hashicorp/hc-install#245</a></li> <li>Update CONTRIBUTING.md to add clean up step by <a href="https://github.com/mutahhir"><code>@mutahhir</code></a> in <a href="https://redirect.github.com/hashicorp/hc-install/pull/246">hashicorp/hc-install#246</a></li> </ul> <h2>New Contributors</h2> <ul> <li><a href="https://github.com/jeanneryan"><code>@jeanneryan</code></a> made their first contribution in <a href="https://redirect.github.com/hashicorp/hc-install/pull/235">hashicorp/hc-install#235</a></li> <li><a href="https://github.com/mutahhir"><code>@mutahhir</code></a> made their first contribution in <a href="https://redirect.github.com/hashicorp/hc-install/pull/246">hashicorp/hc-install#246</a></li> </ul> <p><strong>Full Changelog</strong>: <a href="https://github.com/hashicorp/hc-install/compare/v0.8.0...v0.8.1">https://github.com/hashicorp/hc-install/compare/v0.8.0...v0.8.1</a></p> <h2>v0.8.0</h2> <p>ENHANCEMENTS:</p> <ul> <li>Add retries for HTTP operations by <a href="https://github.com/james0209"><code>@james0209</code></a> in <a href="https://redirect.github.com/hashicorp/hc-install/pull/218">hashicorp/hc-install#218</a></li> <li>Allow <code>LicenseDir</code> field for non-enterprise usage by <a href="https://github.com/james0209"><code>@james0209</code></a> in <a href="https://redirect.github.com/hashicorp/hc-install/pull/214">hashicorp/hc-install#214</a></li> </ul> <p>BUG FIXES:</p> <ul> <li>[fix] include custom url's "path" when creating Archive URL by <a href="https://github.com/james0209"><code>@james0209</code></a> in <a href="https://redirect.github.com/hashicorp/hc-install/pull/234">hashicorp/hc-install#234</a></li> </ul> <p>INTERNAL:</p> <ul> <li>[chore] Remove unused variable by <a href="https://github.com/james0209"><code>@james0209</code></a> in <a href="https://redirect.github.com/hashicorp/hc-install/pull/215">hashicorp/hc-install#215</a></li> <li>build(deps): bump github.com/hashicorp/go-retryablehttp from 0.7.6 to 0.7.7 by <a href="https://github.com/dependabot"><code>@dependabot</code></a> in <a href="https://redirect.github.com/hashicorp/hc-install/pull/221">hashicorp/hc-install#221</a></li> <li>build(deps): bump github.com/hashicorp/go-version from 1.6.0 to 1.7.0 by <a href="https://github.com/dependabot"><code>@dependabot</code></a> in <a href="https://redirect.github.com/hashicorp/hc-install/pull/216">hashicorp/hc-install#216</a></li> <li>build(deps): bump golang.org/x/mod from 0.17.0 to 0.18.0 by <a href="https://github.com/dependabot"><code>@dependabot</code></a> in <a href="https://redirect.github.com/hashicorp/hc-install/pull/223">hashicorp/hc-install#223</a></li> <li>build(deps): bump golang.org/x/mod from 0.18.0 to 0.19.0 by <a href="https://github.com/dependabot"><code>@dependabot</code></a> in <a href="https://redirect.github.com/hashicorp/hc-install/pull/229">hashicorp/hc-install#229</a></li> <li>build(deps): bump hashicorp/action-setup-bob from 2.0.0 to 2.0.3 in the github-actions-backward-compatible group by <a href="https://github.com/dependabot"><code>@dependabot</code></a> in <a href="https://redirect.github.com/hashicorp/hc-install/pull/220">hashicorp/hc-install#220</a></li> <li>build(deps): bump hashicorp/action-setup-bob from 2.0.3 to 2.1.0 in the github-actions-backward-compatible group by <a href="https://github.com/dependabot"><code>@dependabot</code></a> in <a href="https://redirect.github.com/hashicorp/hc-install/pull/222">hashicorp/hc-install#222</a></li> <li>build(deps): bump hashicorp/actions-packaging-linux from 1.7 to 1.8 in the github-actions-backward-compatible group by <a href="https://github.com/dependabot"><code>@dependabot</code></a> in <a href="https://redirect.github.com/hashicorp/hc-install/pull/224">hashicorp/hc-install#224</a></li> <li>build(deps): Bump workflows to latest trusted versions by <a href="https://github.com/hashicorp-tsccr"><code>@hashicorp-tsccr</code></a> in <a href="https://redirect.github.com/hashicorp/hc-install/pull/219">hashicorp/hc-install#219</a></li> <li>build(deps): Bump workflows to latest trusted versions by <a href="https://github.com/hashicorp-tsccr"><code>@hashicorp-tsccr</code></a> in <a href="https://redirect.github.com/hashicorp/hc-install/pull/226">hashicorp/hc-install#226</a></li> </ul> <!-- raw HTML omitted --> </blockquote> <p>... (truncated)</p> </details> <details> <summary>Commits</summary> <ul> <li><a href=" |
|
|
dependabot[bot]
|
875b112f80
|
Bump golang.org/x/mod from 0.20.0 to 0.21.0 (#1758)
Bumps [golang.org/x/mod](https://github.com/golang/mod) from 0.20.0 to 0.21.0. <details> <summary>Commits</summary> <ul> <li><a href=" |
|
|
shreyas-goenka
|
495040e4cd
|
Modify SetLocation test utility to take full locations as argument (#1788)
I plan to use this in https://github.com/databricks/cli/pull/1780, to set the line and column numbers as well for the locations. gopatch file used: ``` @@ var x expression var y expression var z expression @@ -bundletest.SetLocation(x, y, z) +bundletest.SetLocation(x, y, []dyn.Location{{File: z}}) ``` |
|
|
Pieter Noordhuis
|
7f1121d8d8
|
Pin Go toolchain to 1.22.7 (#1790)
## Changes Relates to https://github.com/databricks/cli/pull/1758. More information about toolchains: * https://go.dev/blog/toolchain * https://go.dev/doc/toolchain We need to specify the toolchain as we need to bump Go to 1.22.0 for the `mod` upgrade and want to use the latest toolchain on the 1.22 series. ## Tests The previous release was made with Go 1.22.7 so we should continue to use it. |
|
|
shreyas-goenka
|
a4ba0bbe9f
|
Add sub-extension to resource files in built-in templates (#1777)
## Changes We want to encourage a pattern of only specifying a single resource in a YAML file when an `.<resource-type>.yml` (like `.job.yml`) is used. This convention could allow us to bijectively map a resource YAML file to it's corresponding resource in the Databricks workspace. This PR simply makes the built-in templates compliant to this format. ## Tests Existing tests. |
|
|
Andrew Nester
|
b3a3071086
|
Fixed full variable override detection (#1787)
## Changes Fixes #1786 ## Tests All valid override combinations are added as test cases |
|
Gleb Kanterov
|
3d9decdda9
|
Add JobTaskClusterSpec validate mutator (#1784)
## Changes
Add JobTaskClusterSpec validate mutator. It catches the case when tasks
don't which cluster to use.
For example, we can get this error with minor modifications to
`default-python` template:
```yaml
tasks:
- task_key: python_file_task
spark_python_task:
python_file: ../src/my_project_10/main.py
```
```
% databricks bundle validate
Error: Missing required cluster or environment settings
at resources.jobs.my_project_10_job.tasks[0]
in resources/my_project_10_job.yml:17:11
Task "print_github_stars" requires a cluster or an environment to run.
Specify one of the following fields: job_cluster_key, environment_key, existing_cluster_id, new_cluster.
```
We implicitly rely on "one of" validation, which does not exist. Many
bundle fields can't co-exist, for instance, specifying:
`JobTask.{existing_cluster_id,job_cluster_key}`, `Library.{whl,pypi}`,
`JobTask.{notebook_task,python_wheel_task}`, etc.
## Tests
Unit tests
---------
Co-authored-by: Pieter Noordhuis <pcnoordhuis@gmail.com>
|
|
|
Gleb Kanterov
|
490259a14a
|
Refactor jobs path translation (#1782)
## Changes Extract package for other modules to transform different kinds of paths in job resources. ## Tests Unit tests |
|
|
shreyas-goenka
|
0cc35ca056
|
Assert tokens are redacted in origin URL when username is not specified (#1785)
TSIA |
|
|
Andrew Nester
|
56ed9bebf3
|
Added support for creating all-purpose clusters (#1698)
## Changes
Added support for creating all-purpose clusters
Example of configuration
```
bundle:
name: clusters
resources:
clusters:
test_cluster:
cluster_name: "Test Cluster"
num_workers: 2
node_type_id: "i3.xlarge"
autoscale:
min_workers: 2
max_workers: 7
spark_version: "13.3.x-scala2.12"
spark_conf:
"spark.executor.memory": "2g"
jobs:
test_job:
name: "Test Job"
tasks:
- task_key: test_task
existing_cluster_id: ${resources.clusters.test_cluster.id}
notebook_task:
notebook_path: "./src/test.py"
targets:
development:
mode: development
compute_id: ${resources.clusters.test_cluster.id}
```
## Tests
Added unit, config and E2E tests
|
|
Ilia Babanov
|
ac80d3dfcb
|
Add verbose flag to the "bundle deploy" command (#1774)
## Changes - Extract sync output logic from `cmd/sync` into `lib/sync` - Add hidden `verbose` flag to the `bundle deploy` command, it's false by default and hidden from the `--help` output - Pass output handler to the `deploy/files/upload` mutator if the verbose option is true The was an idea to use in-place output overriding each past file sync event in the output, bit that wont work for the extension, since it doesn't display deploy logs in the terminal. Example output: ``` ~/tmp/defpy: ~/cli/cli bundle deploy --sync-progress Building defpy... Uploading defpy-0.0.1+20240917.112755-py3-none-any.whl... Uploading bundle files to /Users/ilia.babanov@databricks.com/.bundle/defpy/dev/files... Action: PUT: requirements-dev.txt, resources/defpy_pipeline.yml, pytest.ini, src/defpy/main.py, src/defpy/__init__.py, src/dlt_pipeline.ipynb, tests/main_test.py, src/notebook.ipynb, setup.py, resources/defpy_job.yml, .vscode/extensions.json, .vscode/settings.json, fixtures/.gitkeep, .vscode/__builtins__.pyi, README.md, .gitignore, databricks.yml Uploaded tests Uploaded resources Uploaded fixtures Uploaded .vscode Uploaded src/defpy Uploaded requirements-dev.txt Uploaded .gitignore Uploaded fixtures/.gitkeep Uploaded src/defpy/__init__.py Uploaded databricks.yml Uploaded README.md Uploaded setup.py Uploaded .vscode/__builtins__.pyi Uploaded .vscode/extensions.json Uploaded src/dlt_pipeline.ipynb Uploaded .vscode/settings.json Uploaded resources/defpy_job.yml Uploaded pytest.ini Uploaded src/defpy/main.py Uploaded tests/main_test.py Uploaded resources/defpy_pipeline.yml Uploaded src/notebook.ipynb Initial Sync Complete Deploying resources... Updating deployment state... Deployment complete! ``` Output example in the extension: <img width="1843" alt="Screenshot 2024-09-19 at 11 07 48" src="https://github.com/user-attachments/assets/0fafd095-cdc6-44b8-b482-27a38ada0330"> ## Tests Manually for the `sync` and `bundle deploy` commands + vscode extension sync and deploy flows |
|
|
Lennart Kats (databricks)
|
7665c639bd
|
Use Unity Catalog for pipelines in the default-python template (#1766)
## Summary
Enables Unity Catalog for pipelines in the default template. Pipelines
will default to non-Unity Catalog pipelines if a catalog is not
specified.
*Small caveat*: there are cases where admins lock down the default
catalog of a workspace and don't allow the creation of a new schema
there. If that happens, the pipeline would fail at runtime with a clear
error indicating what happened. ("PERMISSION_DENIED: User does not have
CREATE SCHEMA on Catalog 'main'."). I've seen this with an internal
Databricks workspace, where creating new non-UC schemas wasn't locked
down, but creation in the `main` was.
## Testing
- Validated on a non-UC + UC workspace. The catalog selection logic here
is the same as applied for the SQL templates.
|
|
|
Lennart Kats (databricks)
|
6c57683dc6
|
Reduce time until the prompt is shown for bundle run (#1727)
## Summary Makes the `databricks bundle run` command use local state before showing the menu prompt, which makes it show more quickly. For large/busy workspaces this means the prompt can show 2-3 seconds earlier. |
|
|
Andrew Nester
|
cf989a7e10
|
Upgrade to TF provider 1.52 (#1781)
## Changes Upgrade to TF provider 1.52 We also temporarily skip generating plugin framework structs to unblock upgrade as generation does not work yet and need to be fixed separately |
|
|
Andrew Nester
|
e2c1d51d84
|
[Release] Release v0.228.1 (#1778)
Bundles: * Added listing cluster filtering for cluster lookups ([#1754](https://github.com/databricks/cli/pull/1754)). * Expand library globs relative to the sync root ([#1756](https://github.com/databricks/cli/pull/1756)). * Fixed generated YAML missing 'default' for empty values ([#1765](https://github.com/databricks/cli/pull/1765)). * Use periodic triggers in all templates ([#1739](https://github.com/databricks/cli/pull/1739)). * Use the friendly name of service principals when shortening their name ([#1770](https://github.com/databricks/cli/pull/1770)). * Fixed detecting full syntax variable override which includes type field ([#1775](https://github.com/databricks/cli/pull/1775)). Internal: * Pass copy of `dyn.Path` to callback function ([#1747](https://github.com/databricks/cli/pull/1747)). * Make bundle JSON schema modular with `$defs` ([#1700](https://github.com/databricks/cli/pull/1700)). * Alias variables block in the `Target` struct ([#1748](https://github.com/databricks/cli/pull/1748)). * Add end to end integration tests for bundle JSON schema ([#1726](https://github.com/databricks/cli/pull/1726)). * Fix artifact upload integration tests ([#1767](https://github.com/databricks/cli/pull/1767)). API Changes: * Added `databricks quality-monitors regenerate-dashboard` command. OpenAPI commit d05898328669a3f8ab0c2ecee37db2673d3ea3f7 (2024-09-04) Dependency updates: * Bump golang.org/x/term from 0.23.0 to 0.24.0 ([#1757](https://github.com/databricks/cli/pull/1757)). * Bump golang.org/x/oauth2 from 0.22.0 to 0.23.0 ([#1761](https://github.com/databricks/cli/pull/1761)). * Bump golang.org/x/text from 0.17.0 to 0.18.0 ([#1759](https://github.com/databricks/cli/pull/1759)). * Bump github.com/databricks/databricks-sdk-go from 0.45.0 to 0.46.0 ([#1760](https://github.com/databricks/cli/pull/1760)). |
|
|
Andrew Nester
|
bcab6ca37b
|
Fixed detecting full syntax variable override which includes type field (#1775)
## Changes Fixes #1773 ## Tests Confirmed manually |
|
|
Lennart Kats (databricks)
|
e220f9ddd6
|
Use the friendly name of service principals when shortening their name (#1770)
## Summary Use the friendly name of service principals when shortening their name. This change is helpful for the prefix in development mode. Instead of adding a prefix like `[dev 1706906c-c0a2-4c25-9f57-3a7aa3cb8123]`, we'll prefix like `[dev my_principal]`. |
|
|
Lennart Kats (databricks)
|
f2dee890b8
|
Use periodic triggers in all templates (#1739)
## Summary Simplifies template by using the periodic trigger syntax instead of the cron schedule syntax. Periodic triggers are simpler to configure, simpler to read, and make sure that workloads are spread out through the day. We only recommend cron syntax for advanced cases or when more control is needed. ## Testing * Templates validation via unit tests * Manual validation that the new triggers work as expected in dev/prod |
|
|
Pieter Noordhuis
|
fb077a85d2
|
Fix artifact upload integration tests (#1767)
## Changes I didn't run integration tests on #1756. ## Tests Manually confirmed integration tests pass. |
|
|
Andrew Nester
|
66307134c1
|
Fixed generated YAML missing 'default' for empty values (#1765)
## Changes Fixed generated YAML missing 'default' for empty values ## Tests Added unit test |
|
|
shreyas-goenka
|
c61358407f
|
Add end to end integration tests for bundle JSON schema (#1726) | |
|
shreyas-goenka
|
5d2c0e3885
|
Alias variables block in the `Target` struct (#1748)
## Changes
This PR aliases and overrides the schema associated with the variables
block in `target` to allow for directly specifying a variable value in
the JSON schema (without an levels of nesting). This is needed because
this direct value is resolved by dynamically parsing the configuration
tree.
|
|
|
shreyas-goenka
|
28b39cd3f7
|
Make bundle JSON schema modular with `$defs` (#1700)
## Changes This PR makes sweeping changes to the way we generate and test the bundle JSON schema. The main benefits are: 1. More modular JSON schema. Every definition in the schema now is one level deep and points to references instead of inlining the entire schema for a field. This unblocks PyDABs from taking a dependency on the JSON schema. 2. Generate the JSON schema during CLI code generation. Directly stream it instead of computing it at runtime whenever a user calls `databricks bundle schema`. This is nice because we no longer need to embed a partial OpenAPI spec in the CLI. Down the line, we can add a `Schema()` method to every struct in the Databricks Go SDK and remove the dependency on the OpenAPI spec altogether. It'll become more important once we decouple Go SDK structs and methods from the underlying APIs. 3. Add enum values for Go SDK fields in the JSON schema. Better autocompletion and validation for these fields. As a follow-up, we can add enum values for non-Go SDK enums as well (created internal ticket to track). 4. Use "packageName.structName" as a key to read JSON schemas from the OpenAPI spec for Go SDK structs. Before, we would use an unrolled presentation of the JSON schema (stored in `bundle_descriptions.json`), which was complex to parse and include in the final JSON schema output. This also means loading values from the OpenAPI spec for `target` schema works automatically and no longer needs custom code. 5. Support recursive types (eg: `for_each_task`). With us now using $refs everywhere it's trivial to support. 6. Using complex variables would be invalid according to the schema generated before this PR. Now that bug is fixed. In the future adding more custom rules will be easier as well due to the single level nature of the JSON schema. Since this is a complete change of approach in how we generate the JSON schema, there are a few (very minor) regressions worth calling out. 1. We'll lose a few custom descriptions for non Go SDK structs that were a part of `bundle_descriptions.json`. Support for those can be added in the future as a followup. 2. Since now the final JSON schema is a static artefact, we lose some lead time for the signal that JSON schema integration tests are failing. It's okay though since we have a lot of coverage via the existing unit tests. ## Tests Unit tests. End to end tests are being added in this PR: https://github.com/databricks/cli/pull/1726 Previous unit tests were all deleted because they were bloated. Effort was made to make the new unit tests provide (almost) equivalent coverage. |
|
|
dependabot[bot]
|
d3e221a116
|
Bump github.com/databricks/databricks-sdk-go from 0.45.0 to 0.46.0 (#1760)
Bumps [github.com/databricks/databricks-sdk-go](https://github.com/databricks/databricks-sdk-go) from 0.45.0 to 0.46.0. <details> <summary>Release notes</summary> <p><em>Sourced from <a href="https://github.com/databricks/databricks-sdk-go/releases">github.com/databricks/databricks-sdk-go's releases</a>.</em></p> <blockquote> <h2>v0.46.0</h2> <h3>Bug Fixes</h3> <ul> <li>Fail fast when authenticating if host is not configured (<a href="https://redirect.github.com/databricks/databricks-sdk-go/pull/1033">#1033</a>).</li> <li>Improve non-JSON error handling (<a href="https://redirect.github.com/databricks/databricks-sdk-go/pull/1031">#1031</a>).</li> </ul> <h3>Internal Changes</h3> <ul> <li>Add TestAccCreateOboTokenOnAws to flaky test list (<a href="https://redirect.github.com/databricks/databricks-sdk-go/pull/1029">#1029</a>).</li> <li>Add workflows manage integration tests checks (<a href="https://redirect.github.com/databricks/databricks-sdk-go/pull/1032">#1032</a>).</li> <li>Fix TestMwsAccWorkspaces cleanup (<a href="https://redirect.github.com/databricks/databricks-sdk-go/pull/1028">#1028</a>).</li> <li>Improve integration test comment (<a href="https://redirect.github.com/databricks/databricks-sdk-go/pull/1035">#1035</a>).</li> <li>Temporary ignore Metastore test failures (<a href="https://redirect.github.com/databricks/databricks-sdk-go/pull/1027">#1027</a>).</li> <li>Update test to support new accounts (<a href="https://redirect.github.com/databricks/databricks-sdk-go/pull/1026">#1026</a>).</li> <li>Use statuses instead of checks (<a href="https://redirect.github.com/databricks/databricks-sdk-go/pull/1036">#1036</a>).</li> </ul> <h3>API Changes:</h3> <ul> <li>Added <code>RegenerateDashboard</code> method for <a href="https://pkg.go.dev/github.com/databricks/databricks-sdk-go/service/catalog#QualityMonitorsAPI">w.QualityMonitors</a> workspace-level service.</li> <li>Added <a href="https://pkg.go.dev/github.com/databricks/databricks-sdk-go/service/catalog#RegenerateDashboardRequest">catalog.RegenerateDashboardRequest</a> and <a href="https://pkg.go.dev/github.com/databricks/databricks-sdk-go/service/catalog#RegenerateDashboardResponse">catalog.RegenerateDashboardResponse</a>.</li> <li>Added <a href="https://pkg.go.dev/github.com/databricks/databricks-sdk-go/service/jobs#QueueDetails">jobs.QueueDetails</a>, <a href="https://pkg.go.dev/github.com/databricks/databricks-sdk-go/service/jobs#QueueDetailsCodeCode">jobs.QueueDetailsCodeCode</a>, <a href="https://pkg.go.dev/github.com/databricks/databricks-sdk-go/service/jobs#RunLifecycleStateV2State">jobs.RunLifecycleStateV2State</a>, <a href="https://pkg.go.dev/github.com/databricks/databricks-sdk-go/service/jobs#RunStatus">jobs.RunStatus</a>, <a href="https://pkg.go.dev/github.com/databricks/databricks-sdk-go/service/jobs#TerminationCodeCode">jobs.TerminationCodeCode</a>, <a href="https://pkg.go.dev/github.com/databricks/databricks-sdk-go/service/jobs#TerminationDetails">jobs.TerminationDetails</a> and <a href="https://pkg.go.dev/github.com/databricks/databricks-sdk-go/service/jobs#TerminationTypeType">jobs.TerminationTypeType</a>.</li> <li>Added <code>Status</code> field for <a href="https://pkg.go.dev/github.com/databricks/databricks-sdk-go/service/jobs#BaseRun">jobs.BaseRun</a>.</li> <li>Added <code>Status</code> field for <a href="https://pkg.go.dev/github.com/databricks/databricks-sdk-go/service/jobs#RepairHistoryItem">jobs.RepairHistoryItem</a>.</li> <li>Added <code>Status</code> field for <a href="https://pkg.go.dev/github.com/databricks/databricks-sdk-go/service/jobs#Run">jobs.Run</a>.</li> <li>Added <code>Status</code> field for <a href="https://pkg.go.dev/github.com/databricks/databricks-sdk-go/service/jobs#RunTask">jobs.RunTask</a>.</li> <li>Added <code>MaxProvisionedThroughput</code> and <code>MinProvisionedThroughput</code> fields for <a href="https://pkg.go.dev/github.com/databricks/databricks-sdk-go/service/serving#ServedModelInput">serving.ServedModelInput</a>.</li> <li>Added <code>ColumnsToSync</code> field for <a href="https://pkg.go.dev/github.com/databricks/databricks-sdk-go/service/vectorsearch#DeltaSyncVectorIndexSpecRequest">vectorsearch.DeltaSyncVectorIndexSpecRequest</a>.</li> <li>Changed <code>WorkloadSize</code> field for <a href="https://pkg.go.dev/github.com/databricks/databricks-sdk-go/service/serving#ServedModelInput">serving.ServedModelInput</a> to no longer be required.</li> </ul> <p>OpenAPI SHA: d05898328669a3f8ab0c2ecee37db2673d3ea3f7, Date: 2024-09-04</p> </blockquote> </details> <details> <summary>Changelog</summary> <p><em>Sourced from <a href="https://github.com/databricks/databricks-sdk-go/blob/main/CHANGELOG.md">github.com/databricks/databricks-sdk-go's changelog</a>.</em></p> <blockquote> <h2>[Release] Release v0.46.0</h2> <h3>Bug Fixes</h3> <ul> <li>Fail fast when authenticating if host is not configured (<a href="https://redirect.github.com/databricks/databricks-sdk-go/pull/1033">#1033</a>).</li> <li>Improve non-JSON error handling (<a href="https://redirect.github.com/databricks/databricks-sdk-go/pull/1031">#1031</a>).</li> </ul> <h3>Internal Changes</h3> <ul> <li>Add TestAccCreateOboTokenOnAws to flaky test list (<a href="https://redirect.github.com/databricks/databricks-sdk-go/pull/1029">#1029</a>).</li> <li>Add workflows manage integration tests checks (<a href="https://redirect.github.com/databricks/databricks-sdk-go/pull/1032">#1032</a>).</li> <li>Fix TestMwsAccWorkspaces cleanup (<a href="https://redirect.github.com/databricks/databricks-sdk-go/pull/1028">#1028</a>).</li> <li>Improve integration test comment (<a href="https://redirect.github.com/databricks/databricks-sdk-go/pull/1035">#1035</a>).</li> <li>Temporary ignore Metastore test failures (<a href="https://redirect.github.com/databricks/databricks-sdk-go/pull/1027">#1027</a>).</li> <li>Update test to support new accounts (<a href="https://redirect.github.com/databricks/databricks-sdk-go/pull/1026">#1026</a>).</li> <li>Use statuses instead of checks (<a href="https://redirect.github.com/databricks/databricks-sdk-go/pull/1036">#1036</a>).</li> </ul> <h3>API Changes:</h3> <ul> <li>Added <code>RegenerateDashboard</code> method for <a href="https://pkg.go.dev/github.com/databricks/databricks-sdk-go/service/catalog#QualityMonitorsAPI">w.QualityMonitors</a> workspace-level service.</li> <li>Added <a href="https://pkg.go.dev/github.com/databricks/databricks-sdk-go/service/catalog#RegenerateDashboardRequest">catalog.RegenerateDashboardRequest</a> and <a href="https://pkg.go.dev/github.com/databricks/databricks-sdk-go/service/catalog#RegenerateDashboardResponse">catalog.RegenerateDashboardResponse</a>.</li> <li>Added <a href="https://pkg.go.dev/github.com/databricks/databricks-sdk-go/service/jobs#QueueDetails">jobs.QueueDetails</a>, <a href="https://pkg.go.dev/github.com/databricks/databricks-sdk-go/service/jobs#QueueDetailsCodeCode">jobs.QueueDetailsCodeCode</a>, <a href="https://pkg.go.dev/github.com/databricks/databricks-sdk-go/service/jobs#RunLifecycleStateV2State">jobs.RunLifecycleStateV2State</a>, <a href="https://pkg.go.dev/github.com/databricks/databricks-sdk-go/service/jobs#RunStatus">jobs.RunStatus</a>, <a href="https://pkg.go.dev/github.com/databricks/databricks-sdk-go/service/jobs#TerminationCodeCode">jobs.TerminationCodeCode</a>, <a href="https://pkg.go.dev/github.com/databricks/databricks-sdk-go/service/jobs#TerminationDetails">jobs.TerminationDetails</a> and <a href="https://pkg.go.dev/github.com/databricks/databricks-sdk-go/service/jobs#TerminationTypeType">jobs.TerminationTypeType</a>.</li> <li>Added <code>Status</code> field for <a href="https://pkg.go.dev/github.com/databricks/databricks-sdk-go/service/jobs#BaseRun">jobs.BaseRun</a>.</li> <li>Added <code>Status</code> field for <a href="https://pkg.go.dev/github.com/databricks/databricks-sdk-go/service/jobs#RepairHistoryItem">jobs.RepairHistoryItem</a>.</li> <li>Added <code>Status</code> field for <a href="https://pkg.go.dev/github.com/databricks/databricks-sdk-go/service/jobs#Run">jobs.Run</a>.</li> <li>Added <code>Status</code> field for <a href="https://pkg.go.dev/github.com/databricks/databricks-sdk-go/service/jobs#RunTask">jobs.RunTask</a>.</li> <li>Added <code>MaxProvisionedThroughput</code> and <code>MinProvisionedThroughput</code> fields for <a href="https://pkg.go.dev/github.com/databricks/databricks-sdk-go/service/serving#ServedModelInput">serving.ServedModelInput</a>.</li> <li>Added <code>ColumnsToSync</code> field for <a href="https://pkg.go.dev/github.com/databricks/databricks-sdk-go/service/vectorsearch#DeltaSyncVectorIndexSpecRequest">vectorsearch.DeltaSyncVectorIndexSpecRequest</a>.</li> <li>Changed <code>WorkloadSize</code> field for <a href="https://pkg.go.dev/github.com/databricks/databricks-sdk-go/service/serving#ServedModelInput">serving.ServedModelInput</a> to no longer be required.</li> </ul> <p>OpenAPI SHA: d05898328669a3f8ab0c2ecee37db2673d3ea3f7, Date: 2024-09-04</p> </blockquote> </details> <details> <summary>Commits</summary> <ul> <li><a href=" |