Commit Graph

636 Commits

Author SHA1 Message Date
shreyas-goenka ce9c9148c9
Regenerate bundle resource structs from latest terraform provider (#633)
## Changes
This PR:
1. Regenerates the terraform provider structs based off the latest
terraform provider version: 1.22.0
2. Adds a debug launch configuration for regenerating the schema

## Tests
Existing unit tests
2023-08-03 11:20:30 +00:00
Kartik Gupta 3140a8feef
Initialise a empty default bundle if BUNDLE_ROOT and DATABRICKS_BUNDLE_INCLUDES env vars are present (#604)
## Changes
<!-- Summary of your changes that are easy to understand -->

## Tests
<!-- How is this tested? -->
2023-08-02 17:22:47 +00:00
Kartik Gupta 31b178ad6c
Add DATABRICKS_BUNDLE_INCLUDE_PATHS to specify include paths through env vars (#591)
## Changes
* This PR adds `DATABRICKS_BUNDLE_INCLUDE_PATHS` environment variable,
so that we can specify including bundle config files, which we do not
want to commit. These could potentially be local dev overrides or
overrides by our tools - like the VS Code extension
* We always add these include paths to the "include" field. 
## Tests
* [x] Unit tests
2023-08-02 10:18:19 +00:00
Miles Yucht d9ab465ff9
Infer host from profile during login (#629)
## Changes
A pretty annoying part of the current CLI experience is that when
logging in with `databricks auth login`, you always need to type the
name of the host. This seems unnecessary if you have already logged into
a host before, since the CLI can read the previous host from your
`.databrickscfg` file.

This change handles this case by setting the host if unspecified to the
host in the corresponding profile. Combined with autocomplete, this
makes the login process simple:
```
databricks auth login --profile prof<tab><enter>
```

## Tests
Logged in to an existing profile by running the above command (but for a
real profile I had).
2023-08-02 09:43:42 +00:00
shreyas-goenka 5df8935de4
Add JSON schema validation for input template parameters (#598)
## Changes

This PR:
1. Adds code for reading template configs and validating them against a
JSON schema.
2. Moves the json schema struct in `bundle/schema` to a separate library
package. This struct is now reused for validating template configs.

## Tests
Unit tests
2023-08-01 14:09:27 +00:00
shreyas-goenka fc8729d162
Only treat files with .tmpl extension as templates (#594)
## Changes
In a world before this PR, all files would be treated as `go text
templates`, making the content in these files quake in fear since they
would be executed (as a template).

This PR makes it so that only files with the `.tmpl` extension are
understood to be templates. This is useful for avoiding ambiguity in
cases like where a binary file could be interpreted as a go text
template otherwise.

In order to do so, we introduce the `copyFile` struct which does a copy
of the source file from the template without loading it into memory.

## Tests
Unit tests
2023-08-01 13:43:27 +00:00
Miles Yucht bb415ce6bb
Bump OpenAPI specification & Go SDK Version (#624)
## Changes
Bump the OpenAPI specification and Go SDK version to the latest version.

## Tests
<!-- How is this tested? -->
2023-08-01 12:33:19 +00:00
shreyas-goenka 5a6177127f
Fix failing fs mkdir test on azure (#627)
Regex check was missing a "." character and adding it in fixes the test.
The test now passes on all three cloud providers
2023-07-31 10:38:42 +00:00
Lennart Kats (databricks) 433f401c83
Add validation for Git settings in bundles (#578)
## Changes

This checks whether the Git settings are consistent with the actual Git
state of a source directory.

(This PR adds to https://github.com/databricks/cli/pull/577.) 

Previously, we would silently let users configure their Git branch to
e.g. `main` and deploy with that metadata even if they were actually on
a different branch.

With these changes, the following config would result in an error when
deployed from any other branch than `main`:

```
bundle:
  name: example

workspace:
  git:
    branch: main

environments:
  ...
```

> not on the right Git branch:
>   expected according to configuration: main
>   actual: my-feature-branch

It's not very useful to set the same branch for all environments,
though. For development, it's better to just let the CLI auto-detect the
right branch. Therefore, it's now possible to set the branch just for a
single environment:

```
bundle:
  name: example 2

environments:
  development:
    default: true

  production:
    # production can only be deployed from the 'main' branch
    git:
      branch: main
```

Adding to that, the `mode: production` option actually checks that users
explicitly set the Git branch as seen above. Setting that branch helps
avoid mistakes, where someone accidentally deploys to production from
the wrong branch. (I could see us offering an escape hatch for that in
the future.)

# Testing

Manual testing to validate the experience and error messages. Automated
unit tests.

---------

Co-authored-by: Fabian Jakobs <fabian.jakobs@databricks.com>
2023-07-30 12:44:33 +00:00
Lennart Kats (databricks) d55652be07
Extend deployment mode support (#577)
## Changes

This adds `mode: production` option. This mode doesn't do any
transformations but verifies that an environment is configured correctly
for production:

```
environments:
  prod:
    mode: production

    # paths should not be scoped to a user (unless a service principal is used)
    root_path: /Shared/non_user_path/...

    # run_as and permissions should be set at the resource level (or at the top level when that is implemented)
    run_as:
      user_name: Alice
    permissions:
    - level: CAN_MANAGE
      user_name: Alice
```

Additionally, this extends the existing `mode: development` option,
* now prefixing deployed assets with `[dev your.user]` instead of just
`[dev`]
* validating that development deployments _are_ scoped to a user

## Related

https://github.com/databricks/cli/pull/578/files (in draft)

## Tests

Manual testing to validate the experience, error messages, and
functionality with all resource types. Automated unit tests.

---------

Co-authored-by: Fabian Jakobs <fabian.jakobs@databricks.com>
2023-07-30 07:19:49 +00:00
Pieter Noordhuis 5deca8d7d8
Remove workflow to publish binaries to S3 (#622)
## Changes

Binaries are published through GitHub releases now.
2023-07-27 14:19:50 +00:00
Pieter Noordhuis 28bfc0c73d
Remove push to main trigger for build workflow (#621)
## Changes

Commits going through the merge queue are tested there using their final
SHA as if they were already in main. The push-to-main trigger therefore
duplicates the builds that were already triggered from the merge queue.

## Tests

![Screenshot 2023-07-27 at 15 37
17](https://github.com/databricks/cli/assets/9845/ff7af5dd-0d2c-48c2-89b2-7ecf3d121071)
2023-07-27 14:19:40 +00:00
shreyas-goenka 2f4bf844fc
Fix git clone integration test for non-existing repo (#610)
## Changes
This PR changes the integration test to just check an error is returned
rather than asserting specific text is present in the error. This is
required because the error returned can be different based on whether
git ssh keys have been setup.
2023-07-27 13:51:57 +00:00
shreyas-goenka e4b66833dd
Fix mkdir integration test on GCP (#620)
GCP returns a different error here incase of a conflict. This PR fixes
the test.
2023-07-27 13:24:34 +00:00
Miles Yucht 3697dfcb51
Release v0.202.0 (#619)
Breaking Change:
* Require include glob patterns to be explicitly defined
([#602](https://github.com/databricks/cli/pull/602)).

Bundles:
* Add support for more SDK config options
([#587](https://github.com/databricks/cli/pull/587)).
* Add template renderer for Databricks templates
([#589](https://github.com/databricks/cli/pull/589)).
* Fix formatting in renderer.go
([#593](https://github.com/databricks/cli/pull/593)).
* Fixed python wheel test
([#608](https://github.com/databricks/cli/pull/608)).
* Auto detect Python wheel packages and infer build command
([#603](https://github.com/databricks/cli/pull/603)).
* Added support for artifacts building for bundles
([#583](https://github.com/databricks/cli/pull/583)).
* Add support for cloning repositories
([#544](https://github.com/databricks/cli/pull/544)).
* Add regexp compile helper function for templates
([#601](https://github.com/databricks/cli/pull/601)).
* Add unit test that raw strings are printed as is
([#599](https://github.com/databricks/cli/pull/599)).

Internal:
* Fix tests under ./cmd/configure if DATABRICKS_TOKEN is set
([#605](https://github.com/databricks/cli/pull/605)).
* Remove dependency on global state in generated commands
([#595](https://github.com/databricks/cli/pull/595)).
* Remove dependency on global state for the root command
([#606](https://github.com/databricks/cli/pull/606)).
* Add merge_group trigger for build
([#612](https://github.com/databricks/cli/pull/612)).
* Added support for build command chaining and error on missing wheel
([#607](https://github.com/databricks/cli/pull/607)).
* Add TestAcc prefix to filer test and fix any failing tests
([#611](https://github.com/databricks/cli/pull/611)).
* Add url parse helper function for templates
([#600](https://github.com/databricks/cli/pull/600)).
* Remove dependency on global state for remaining commands
([#613](https://github.com/databricks/cli/pull/613)).
* Update CHANGELOG template
([#588](https://github.com/databricks/cli/pull/588)).
2023-07-27 13:23:55 +00:00
Pieter Noordhuis e3a181b03d
Update CHANGELOG template (#588)
Made the recent release reflect this template.

---------

Co-authored-by: Miles Yucht <miles@databricks.com>
2023-07-27 10:47:38 +00:00
Pieter Noordhuis bee7a16cb0
Remove dependency on global state for remaining commands (#613)
## Changes

This removes the remaining dependency on global state and unblocks work
to parallelize integration tests. As is, we can already uncomment an
integration test that had to be skipped because of other tests tainting
global state. This is no longer an issue.

Also see #595 and #606.

## Tests

* Unit and integration tests pass.
* Manually confirmed the help output is the same.
2023-07-27 10:03:08 +00:00
shreyas-goenka ed972f7ae0
Add url parse helper function for templates (#600)
## Tests
unit test
2023-07-27 09:51:31 +00:00
shreyas-goenka 8ffff241fe
Add TestAcc prefix to filer test and fix any failing tests (#611)
## Changes
Fs integration tests were not running on our nightlies before because
the nightlies only run tests with the `TestAcc` prefix. A couple of them
were also broken!

This PR fixes the tests and adds the prefix to all fs integration tests.

As a followup we can automate the check for this prefix.

## Tested
Fs tests are green and pass on both azure and aws
2023-07-26 13:03:10 +00:00
Andrew Nester 12bba17743
Added support for build command chaining and error on missing wheel (#607)
## Changes
Added support for build command chaining and error on missing wheel
2023-07-26 12:58:52 +00:00
Pieter Noordhuis 8cdf7284f8
Add merge_group trigger for build (#612)
## Changes

Also see [GitHub
docs](https://docs.github.com/en/repositories/configuring-branches-and-merges-in-your-repository/configuring-pull-request-merges/managing-a-merge-queue#triggering-merge-group-checks-with-github-actions).

## Tests

n/a
2023-07-26 12:48:05 +00:00
Pieter Noordhuis ec892aa11c
Remove dependency on global state for the root command (#606)
## Changes

This change is another step towards a CLI without globals. Also see #595.

The flags for the root command are now encapsulated in struct types.

## Tests

Unit tests pass.
2023-07-26 11:17:09 +00:00
Andrew Nester cfff140815
Auto detect Python wheel packages and infer build command (#603) 2023-07-26 10:07:26 +00:00
Pieter Noordhuis f0ad28ab62
Fix tests under ./cmd/configure if DATABRICKS_TOKEN is set (#605)
## Changes

The assertions would fail because `DATABRICKS_TOKEN` overrides a token
set in the profile.

## Tests

Tests now pass if `DATABRICKS_TOKEN` is set.
2023-07-26 09:37:18 +00:00
Andrew Nester 5e0a096722
Fixed python wheel test (#608)
## Changes
Fixed python wheel test

## Tests
<!-- How is this tested? -->
2023-07-26 11:02:17 +02:00
Shreyas Goenka 1d21d3cfd3
Revert "Fix git clone integration test for non-existing repo"
This reverts commit 7a8d413a4b.
2023-07-26 10:37:17 +02:00
Shreyas Goenka 7a8d413a4b
Fix git clone integration test for non-existing repo 2023-07-26 10:36:49 +02:00
Pieter Noordhuis 3fa400f00f
Remove dependency on global state in generated commands (#595)
## Changes

Generated commands relied on global variables for flags and request
payloads. This is difficult to test if a sequence of tests tries to run
the same command with various arguments because the global state causes
test interference. Moreover, it is impossible to run tests in parallel.

This change modifies the approach and turns every command group and
command itself into a function that returns a `*cobra.Command`. All
flags and request payloads are variables scoped to the command's
initialization function. This means it is possible to construct
independent copies of the CLI structure and fixes the test isolation
issue.

The scope of this change is only the generated commands. The other
commands will be changed accordingly in subsequent changes.

## Tests

Unit and integration tests pass.
2023-07-25 20:19:07 +02:00
shreyas-goenka 34f196bb4e
Add unit test that raw strings are printed as is (#599)
## Changes
Add unit test that raw strings are printed as is. This method is useful
to print text that would otherwise be interpreted a go text template.
2023-07-25 17:18:43 +02:00
shreyas-goenka 47640b8b94
Add regexp compile helper function for templates (#601)
## Tests
unit test
2023-07-25 16:42:53 +02:00
shreyas-goenka 8fdc0fec81
Add support for cloning repositories (#544)
## Changes
Adds support for cloning public and private github repositories for
databricks templates

## Tests
Integration tests
2023-07-25 15:36:20 +02:00
Andrew Nester 9a88fa602d
Added support for artifacts building for bundles (#583)
## Changes
Added support for artifacts building for bundles. 

Now it allows to specify `artifacts` block in bundle.yml and define a
resource (at the moment Python wheel) to be build and uploaded during
`bundle deploy`

Built artifact will be automatically attached to corresponding job task
or pipeline where it's used as a library

Follow-ups:
1. If artifact is used in job or pipeline, but not found in the config,
try to infer and build it anyway
2. If build command is not provided for Python wheel artifact, infer it
2023-07-25 13:35:08 +02:00
shreyas-goenka fa37449f1f
Require include glob patterns to be explicitly defined (#602)
## Changes
Before this PR we would load all yaml files matching * and \*/\*.yml
files as bundle configurations. This was problematic since this would
also load yaml files that were not meant to be a part of the bundle

## Tests
Manually, now files are no longer included unless manually specified
2023-07-25 10:00:46 +02:00
shreyas-goenka 13731e144c
Fix formatting in renderer.go (#593)
## Changes
Due to a bug in Github UI, https://github.com/databricks/cli/pull/589
got merged without passing the go/fmt formatting checks

This PR fixes the formatting which breaks the PR checks
2023-07-21 11:23:47 +02:00
shreyas-goenka 02dbac7b8a
Add template renderer for Databricks templates (#589)
## Changes
This PR adds the renderer struct, which is a walker that traverses
templates and generates projects from them

## Tests
Unit tests
2023-07-21 10:59:02 +02:00
Fabian Jakobs adab9aa5d7
Add support for more SDK config options (#587)
## Changes
Add support for more SDK config options
2023-07-19 14:06:58 +02:00
Serge Smertin acf292da37
Release v0.201.0 (#586)
* Add development runs
([#522](https://github.com/databricks/cli/pull/522)).
* Support tab completion for profiles
([#572](https://github.com/databricks/cli/pull/572)).
* Correctly use --profile flag passed for all bundle commands
([#571](https://github.com/databricks/cli/pull/571)).
* Disallow notebooks in paths where files are expected
([#573](https://github.com/databricks/cli/pull/573)).
* Improve auth login experience
([#570](https://github.com/databricks/cli/pull/570)).
* Remove base path checks during sync
([#576](https://github.com/databricks/cli/pull/576)).
* First look for databricks.yml before falling back to bundle.yml
([#580](https://github.com/databricks/cli/pull/580)).
* Integrate with auto-release infra
([#581](https://github.com/databricks/cli/pull/581)).

API Changes:

 * Removed `databricks metastores maintenance` command.
 * Added `databricks metastores enable-optimization` command.
 * Added `databricks tables update` command.
* Changed `databricks account settings delete-personal-compute-setting`
command with new required argument order.
* Changed `databricks account settings read-personal-compute-setting`
command with new required argument order.
 * Added `databricks clean-rooms` command group.

OpenAPI SHA: 850a075ed9758d21a6bc4409506b48c8b9f93ab4, Date: 2023-07-18
Dependency updates:

* Bump golang.org/x/term from 0.9.0 to 0.10.0
([#567](https://github.com/databricks/cli/pull/567)).
* Bump golang.org/x/oauth2 from 0.9.0 to 0.10.0
([#566](https://github.com/databricks/cli/pull/566)).
* Bump golang.org/x/mod from 0.11.0 to 0.12.0
([#568](https://github.com/databricks/cli/pull/568)).
* Bump github.com/databricks/databricks-sdk-go from 0.12.0 to 0.13.0
([#585](https://github.com/databricks/cli/pull/585)).
2023-07-18 17:13:48 +00:00
Serge Smertin ff98096208
Integrate with auto-release infra (#581)
## Changes
- added changelog template
- added `toolchain` to `.codegen.json`

## Tests
none
2023-07-18 17:48:35 +02:00
dependabot[bot] 65d8fe13e9
Bump github.com/databricks/databricks-sdk-go from 0.12.0 to 0.13.0 (#585)
Bumps
[github.com/databricks/databricks-sdk-go](https://github.com/databricks/databricks-sdk-go)
from 0.12.0 to 0.13.0.
<details>
<summary>Release notes</summary>
<p><em>Sourced from <a
href="https://github.com/databricks/databricks-sdk-go/releases">github.com/databricks/databricks-sdk-go's
releases</a>.</em></p>
<blockquote>
<h2>v0.13.0</h2>
<ul>
<li>Add issue templates (<a
href="https://redirect.github.com/databricks/databricks-sdk-go/pull/539">#539</a>).</li>
<li>Added HasRequiredNonBodyField method (<a
href="https://redirect.github.com/databricks/databricks-sdk-go/pull/536">#536</a>).</li>
<li>Make Azure MSI auth account compatible (<a
href="https://redirect.github.com/databricks/databricks-sdk-go/pull/544">#544</a>).</li>
<li>Refactor Handling of Name<!-- raw HTML omitted -->ID Mapping in
OpenAPI Generator (<a
href="https://redirect.github.com/databricks/databricks-sdk-go/pull/547">#547</a>).</li>
<li>Regenerate Go SDK from current OpenAPI Specification (<a
href="https://redirect.github.com/databricks/databricks-sdk-go/pull/549">#549</a>).</li>
<li>Parse Camel Case and Pascal Case Enum Values (<a
href="https://redirect.github.com/databricks/databricks-sdk-go/pull/550">#550</a>).</li>
<li>Prepare for auto-releaser infra (<a
href="https://redirect.github.com/databricks/databricks-sdk-go/pull/554">#554</a>).</li>
<li>Added SCIM Patch Acceptance Tests (<a
href="https://redirect.github.com/databricks/databricks-sdk-go/pull/540">#540</a>).</li>
</ul>
<p>API Changes:</p>
<ul>
<li>Removed <code>Maintenance</code> method for <a
href="https://pkg.go.dev/github.com/databricks/databricks-sdk-go/service/catalog#MetastoresAPI">w.Metastores</a>
workspace-level service.</li>
<li>Added <code>EnableOptimization</code> method for <a
href="https://pkg.go.dev/github.com/databricks/databricks-sdk-go/service/catalog#MetastoresAPI">w.Metastores</a>
workspace-level service.</li>
<li>Added <code>Update</code> method for <a
href="https://pkg.go.dev/github.com/databricks/databricks-sdk-go/service/catalog#TablesAPI">w.Tables</a>
workspace-level service.</li>
<li>Added <code>Force</code> field for <a
href="https://pkg.go.dev/github.com/databricks/databricks-sdk-go/service/catalog#DeleteAccountMetastoreRequest">catalog.DeleteAccountMetastoreRequest</a>.</li>
<li>Added <code>Force</code> field for <a
href="https://pkg.go.dev/github.com/databricks/databricks-sdk-go/service/catalog#DeleteAccountStorageCredentialRequest">catalog.DeleteAccountStorageCredentialRequest</a>.</li>
<li>Removed <a
href="https://pkg.go.dev/github.com/databricks/databricks-sdk-go/service/catalog#UpdateAutoMaintenance">catalog.UpdateAutoMaintenance</a>.</li>
<li>Removed <a
href="https://pkg.go.dev/github.com/databricks/databricks-sdk-go/service/catalog#UpdateAutoMaintenanceResponse">catalog.UpdateAutoMaintenanceResponse</a>.</li>
<li>Added <a
href="https://pkg.go.dev/github.com/databricks/databricks-sdk-go/service/catalog#UpdatePredictiveOptimization">catalog.UpdatePredictiveOptimization</a>.</li>
<li>Added <a
href="https://pkg.go.dev/github.com/databricks/databricks-sdk-go/service/catalog#UpdatePredictiveOptimizationResponse">catalog.UpdatePredictiveOptimizationResponse</a>.</li>
<li>Added <a
href="https://pkg.go.dev/github.com/databricks/databricks-sdk-go/service/catalog#UpdateTableRequest">catalog.UpdateTableRequest</a>.</li>
<li>Added <code>Schema</code> field for <a
href="https://pkg.go.dev/github.com/databricks/databricks-sdk-go/service/iam#PartialUpdate">iam.PartialUpdate</a>.</li>
<li>Added <a
href="https://pkg.go.dev/github.com/databricks/databricks-sdk-go/service/iam#PatchSchema">iam.PatchSchema</a>.</li>
<li>Added <code>TriggerInfo</code> field for <a
href="https://pkg.go.dev/github.com/databricks/databricks-sdk-go/service/jobs#BaseRun">jobs.BaseRun</a>.</li>
<li>Added <code>Health</code> field for <a
href="https://pkg.go.dev/github.com/databricks/databricks-sdk-go/service/jobs#CreateJob">jobs.CreateJob</a>.</li>
<li>Added <code>JobSource</code> field for <a
href="https://pkg.go.dev/github.com/databricks/databricks-sdk-go/service/jobs#GitSource">jobs.GitSource</a>.</li>
<li>Added <code>OnDurationWarningThresholdExceeded</code> field for <a
href="https://pkg.go.dev/github.com/databricks/databricks-sdk-go/service/jobs#JobEmailNotifications">jobs.JobEmailNotifications</a>.</li>
<li>Added <code>Health</code> field for <a
href="https://pkg.go.dev/github.com/databricks/databricks-sdk-go/service/jobs#JobSettings">jobs.JobSettings</a>.</li>
<li>Added <code>TriggerInfo</code> field for <a
href="https://pkg.go.dev/github.com/databricks/databricks-sdk-go/service/jobs#Run">jobs.Run</a>.</li>
<li>Added <code>RunJobOutput</code> field for <a
href="https://pkg.go.dev/github.com/databricks/databricks-sdk-go/service/jobs#RunOutput">jobs.RunOutput</a>.</li>
<li>Added <code>RunJobTask</code> field for <a
href="https://pkg.go.dev/github.com/databricks/databricks-sdk-go/service/jobs#RunTask">jobs.RunTask</a>.</li>
<li>Added <code>EmailNotifications</code> field for <a
href="https://pkg.go.dev/github.com/databricks/databricks-sdk-go/service/jobs#SubmitRun">jobs.SubmitRun</a>.</li>
<li>Added <code>Health</code> field for <a
href="https://pkg.go.dev/github.com/databricks/databricks-sdk-go/service/jobs#SubmitRun">jobs.SubmitRun</a>.</li>
<li>Added <code>EmailNotifications</code> field for <a
href="https://pkg.go.dev/github.com/databricks/databricks-sdk-go/service/jobs#SubmitTask">jobs.SubmitTask</a>.</li>
<li>Added <code>Health</code> field for <a
href="https://pkg.go.dev/github.com/databricks/databricks-sdk-go/service/jobs#SubmitTask">jobs.SubmitTask</a>.</li>
<li>Added <code>NotificationSettings</code> field for <a
href="https://pkg.go.dev/github.com/databricks/databricks-sdk-go/service/jobs#SubmitTask">jobs.SubmitTask</a>.</li>
<li>Added <code>Health</code> field for <a
href="https://pkg.go.dev/github.com/databricks/databricks-sdk-go/service/jobs#Task">jobs.Task</a>.</li>
<li>Added <code>RunJobTask</code> field for <a
href="https://pkg.go.dev/github.com/databricks/databricks-sdk-go/service/jobs#Task">jobs.Task</a>.</li>
<li>Added <code>OnDurationWarningThresholdExceeded</code> field for <a
href="https://pkg.go.dev/github.com/databricks/databricks-sdk-go/service/jobs#TaskEmailNotifications">jobs.TaskEmailNotifications</a>.</li>
<li>Added <code>OnDurationWarningThresholdExceeded</code> field for <a
href="https://pkg.go.dev/github.com/databricks/databricks-sdk-go/service/jobs#WebhookNotifications">jobs.WebhookNotifications</a>.</li>
<li>Added <a
href="https://pkg.go.dev/github.com/databricks/databricks-sdk-go/service/jobs#JobSource">jobs.JobSource</a>.</li>
<li>Added <a
href="https://pkg.go.dev/github.com/databricks/databricks-sdk-go/service/jobs#JobSourceDirtyState">jobs.JobSourceDirtyState</a>.</li>
<li>Added <a
href="https://pkg.go.dev/github.com/databricks/databricks-sdk-go/service/jobs#JobsHealthMetric">jobs.JobsHealthMetric</a>.</li>
<li>Added <a
href="https://pkg.go.dev/github.com/databricks/databricks-sdk-go/service/jobs#JobsHealthOperator">jobs.JobsHealthOperator</a>.</li>
<li>Added <a
href="https://pkg.go.dev/github.com/databricks/databricks-sdk-go/service/jobs#JobsHealthRule">jobs.JobsHealthRule</a>.</li>
<li>Added <a
href="https://pkg.go.dev/github.com/databricks/databricks-sdk-go/service/jobs#JobsHealthRules">jobs.JobsHealthRules</a>.</li>
<li>Added <a
href="https://pkg.go.dev/github.com/databricks/databricks-sdk-go/service/jobs#RunJobOutput">jobs.RunJobOutput</a>.</li>
<li>Added <a
href="https://pkg.go.dev/github.com/databricks/databricks-sdk-go/service/jobs#RunJobTask">jobs.RunJobTask</a>.</li>
</ul>
<!-- raw HTML omitted -->
</blockquote>
<p>... (truncated)</p>
</details>
<details>
<summary>Changelog</summary>
<p><em>Sourced from <a
href="https://github.com/databricks/databricks-sdk-go/blob/main/CHANGELOG.md">github.com/databricks/databricks-sdk-go's
changelog</a>.</em></p>
<blockquote>
<h2>0.13.0</h2>
<ul>
<li>Add issue templates (<a
href="https://redirect.github.com/databricks/databricks-sdk-go/pull/539">#539</a>).</li>
<li>Added HasRequiredNonBodyField method (<a
href="https://redirect.github.com/databricks/databricks-sdk-go/pull/536">#536</a>).</li>
<li>Make Azure MSI auth account compatible (<a
href="https://redirect.github.com/databricks/databricks-sdk-go/pull/544">#544</a>).</li>
<li>Refactor Handling of Name<!-- raw HTML omitted -->ID Mapping in
OpenAPI Generator (<a
href="https://redirect.github.com/databricks/databricks-sdk-go/pull/547">#547</a>).</li>
<li>Regenerate Go SDK from current OpenAPI Specification (<a
href="https://redirect.github.com/databricks/databricks-sdk-go/pull/549">#549</a>).</li>
<li>Parse Camel Case and Pascal Case Enum Values (<a
href="https://redirect.github.com/databricks/databricks-sdk-go/pull/550">#550</a>).</li>
<li>Prepare for auto-releaser infra (<a
href="https://redirect.github.com/databricks/databricks-sdk-go/pull/554">#554</a>).</li>
<li>Added SCIM Patch Acceptance Tests (<a
href="https://redirect.github.com/databricks/databricks-sdk-go/pull/540">#540</a>).</li>
</ul>
<p>API Changes:</p>
<ul>
<li>Removed <code>Maintenance</code> method for <a
href="https://pkg.go.dev/github.com/databricks/databricks-sdk-go/service/catalog#MetastoresAPI">w.Metastores</a>
workspace-level service.</li>
<li>Added <code>EnableOptimization</code> method for <a
href="https://pkg.go.dev/github.com/databricks/databricks-sdk-go/service/catalog#MetastoresAPI">w.Metastores</a>
workspace-level service.</li>
<li>Added <code>Update</code> method for <a
href="https://pkg.go.dev/github.com/databricks/databricks-sdk-go/service/catalog#TablesAPI">w.Tables</a>
workspace-level service.</li>
<li>Added <code>Force</code> field for <a
href="https://pkg.go.dev/github.com/databricks/databricks-sdk-go/service/catalog#DeleteAccountMetastoreRequest">catalog.DeleteAccountMetastoreRequest</a>.</li>
<li>Added <code>Force</code> field for <a
href="https://pkg.go.dev/github.com/databricks/databricks-sdk-go/service/catalog#DeleteAccountStorageCredentialRequest">catalog.DeleteAccountStorageCredentialRequest</a>.</li>
<li>Removed <a
href="https://pkg.go.dev/github.com/databricks/databricks-sdk-go/service/catalog#UpdateAutoMaintenance">catalog.UpdateAutoMaintenance</a>.</li>
<li>Removed <a
href="https://pkg.go.dev/github.com/databricks/databricks-sdk-go/service/catalog#UpdateAutoMaintenanceResponse">catalog.UpdateAutoMaintenanceResponse</a>.</li>
<li>Added <a
href="https://pkg.go.dev/github.com/databricks/databricks-sdk-go/service/catalog#UpdatePredictiveOptimization">catalog.UpdatePredictiveOptimization</a>.</li>
<li>Added <a
href="https://pkg.go.dev/github.com/databricks/databricks-sdk-go/service/catalog#UpdatePredictiveOptimizationResponse">catalog.UpdatePredictiveOptimizationResponse</a>.</li>
<li>Added <a
href="https://pkg.go.dev/github.com/databricks/databricks-sdk-go/service/catalog#UpdateTableRequest">catalog.UpdateTableRequest</a>.</li>
<li>Added <code>Schema</code> field for <a
href="https://pkg.go.dev/github.com/databricks/databricks-sdk-go/service/iam#PartialUpdate">iam.PartialUpdate</a>.</li>
<li>Added <a
href="https://pkg.go.dev/github.com/databricks/databricks-sdk-go/service/iam#PatchSchema">iam.PatchSchema</a>.</li>
<li>Added <code>TriggerInfo</code> field for <a
href="https://pkg.go.dev/github.com/databricks/databricks-sdk-go/service/jobs#BaseRun">jobs.BaseRun</a>.</li>
<li>Added <code>Health</code> field for <a
href="https://pkg.go.dev/github.com/databricks/databricks-sdk-go/service/jobs#CreateJob">jobs.CreateJob</a>.</li>
<li>Added <code>JobSource</code> field for <a
href="https://pkg.go.dev/github.com/databricks/databricks-sdk-go/service/jobs#GitSource">jobs.GitSource</a>.</li>
<li>Added <code>OnDurationWarningThresholdExceeded</code> field for <a
href="https://pkg.go.dev/github.com/databricks/databricks-sdk-go/service/jobs#JobEmailNotifications">jobs.JobEmailNotifications</a>.</li>
<li>Added <code>Health</code> field for <a
href="https://pkg.go.dev/github.com/databricks/databricks-sdk-go/service/jobs#JobSettings">jobs.JobSettings</a>.</li>
<li>Added <code>TriggerInfo</code> field for <a
href="https://pkg.go.dev/github.com/databricks/databricks-sdk-go/service/jobs#Run">jobs.Run</a>.</li>
<li>Added <code>RunJobOutput</code> field for <a
href="https://pkg.go.dev/github.com/databricks/databricks-sdk-go/service/jobs#RunOutput">jobs.RunOutput</a>.</li>
<li>Added <code>RunJobTask</code> field for <a
href="https://pkg.go.dev/github.com/databricks/databricks-sdk-go/service/jobs#RunTask">jobs.RunTask</a>.</li>
<li>Added <code>EmailNotifications</code> field for <a
href="https://pkg.go.dev/github.com/databricks/databricks-sdk-go/service/jobs#SubmitRun">jobs.SubmitRun</a>.</li>
<li>Added <code>Health</code> field for <a
href="https://pkg.go.dev/github.com/databricks/databricks-sdk-go/service/jobs#SubmitRun">jobs.SubmitRun</a>.</li>
<li>Added <code>EmailNotifications</code> field for <a
href="https://pkg.go.dev/github.com/databricks/databricks-sdk-go/service/jobs#SubmitTask">jobs.SubmitTask</a>.</li>
<li>Added <code>Health</code> field for <a
href="https://pkg.go.dev/github.com/databricks/databricks-sdk-go/service/jobs#SubmitTask">jobs.SubmitTask</a>.</li>
<li>Added <code>NotificationSettings</code> field for <a
href="https://pkg.go.dev/github.com/databricks/databricks-sdk-go/service/jobs#SubmitTask">jobs.SubmitTask</a>.</li>
<li>Added <code>Health</code> field for <a
href="https://pkg.go.dev/github.com/databricks/databricks-sdk-go/service/jobs#Task">jobs.Task</a>.</li>
<li>Added <code>RunJobTask</code> field for <a
href="https://pkg.go.dev/github.com/databricks/databricks-sdk-go/service/jobs#Task">jobs.Task</a>.</li>
<li>Added <code>OnDurationWarningThresholdExceeded</code> field for <a
href="https://pkg.go.dev/github.com/databricks/databricks-sdk-go/service/jobs#TaskEmailNotifications">jobs.TaskEmailNotifications</a>.</li>
<li>Added <code>OnDurationWarningThresholdExceeded</code> field for <a
href="https://pkg.go.dev/github.com/databricks/databricks-sdk-go/service/jobs#WebhookNotifications">jobs.WebhookNotifications</a>.</li>
<li>Added <a
href="https://pkg.go.dev/github.com/databricks/databricks-sdk-go/service/jobs#JobSource">jobs.JobSource</a>.</li>
<li>Added <a
href="https://pkg.go.dev/github.com/databricks/databricks-sdk-go/service/jobs#JobSourceDirtyState">jobs.JobSourceDirtyState</a>.</li>
<li>Added <a
href="https://pkg.go.dev/github.com/databricks/databricks-sdk-go/service/jobs#JobsHealthMetric">jobs.JobsHealthMetric</a>.</li>
<li>Added <a
href="https://pkg.go.dev/github.com/databricks/databricks-sdk-go/service/jobs#JobsHealthOperator">jobs.JobsHealthOperator</a>.</li>
<li>Added <a
href="https://pkg.go.dev/github.com/databricks/databricks-sdk-go/service/jobs#JobsHealthRule">jobs.JobsHealthRule</a>.</li>
<li>Added <a
href="https://pkg.go.dev/github.com/databricks/databricks-sdk-go/service/jobs#JobsHealthRules">jobs.JobsHealthRules</a>.</li>
<li>Added <a
href="https://pkg.go.dev/github.com/databricks/databricks-sdk-go/service/jobs#RunJobOutput">jobs.RunJobOutput</a>.</li>
<li>Added <a
href="https://pkg.go.dev/github.com/databricks/databricks-sdk-go/service/jobs#RunJobTask">jobs.RunJobTask</a>.</li>
</ul>
<!-- raw HTML omitted -->
</blockquote>
<p>... (truncated)</p>
</details>
<details>
<summary>Commits</summary>
<ul>
<li><a
href="b4fb746b3b"><code>b4fb746</code></a>
Release v0.13.0 (<a
href="https://redirect.github.com/databricks/databricks-sdk-go/issues/555">#555</a>)</li>
<li><a
href="180c7eea4c"><code>180c7ee</code></a>
Added SCIM Patch Acceptance Tests (<a
href="https://redirect.github.com/databricks/databricks-sdk-go/issues/540">#540</a>)</li>
<li><a
href="546814a272"><code>546814a</code></a>
Prepare for auto-releaser infra (<a
href="https://redirect.github.com/databricks/databricks-sdk-go/issues/554">#554</a>)</li>
<li><a
href="7e680c5ba8"><code>7e680c5</code></a>
Parse Camel Case and Pascal Case Enum Values (<a
href="https://redirect.github.com/databricks/databricks-sdk-go/issues/550">#550</a>)</li>
<li><a
href="e71ece4ccd"><code>e71ece4</code></a>
Bump google.golang.org/api from 0.130.0 to 0.131.0 (<a
href="https://redirect.github.com/databricks/databricks-sdk-go/issues/551">#551</a>)</li>
<li><a
href="3b4492b6d6"><code>3b4492b</code></a>
Regenerate Go SDK from current OpenAPI Specification (<a
href="https://redirect.github.com/databricks/databricks-sdk-go/issues/549">#549</a>)</li>
<li><a
href="f84de6111a"><code>f84de61</code></a>
Refactor Handling of Name&lt;-&gt;ID Mapping in OpenAPI Generator (<a
href="https://redirect.github.com/databricks/databricks-sdk-go/issues/547">#547</a>)</li>
<li><a
href="c37a894872"><code>c37a894</code></a>
Bump google.golang.org/api from 0.129.0 to 0.130.0 (<a
href="https://redirect.github.com/databricks/databricks-sdk-go/issues/542">#542</a>)</li>
<li><a
href="4f2aa38e75"><code>4f2aa38</code></a>
Bump golang.org/x/mod from 0.11.0 to 0.12.0 (<a
href="https://redirect.github.com/databricks/databricks-sdk-go/issues/541">#541</a>)</li>
<li><a
href="e80f6e16ff"><code>e80f6e1</code></a>
Make Azure MSI auth account compatible (<a
href="https://redirect.github.com/databricks/databricks-sdk-go/issues/544">#544</a>)</li>
<li>Additional commits viewable in <a
href="https://github.com/databricks/databricks-sdk-go/compare/v0.12.0...v0.13.0">compare
view</a></li>
</ul>
</details>
<br />


[![Dependabot compatibility
score](https://dependabot-badges.githubapp.com/badges/compatibility_score?dependency-name=github.com/databricks/databricks-sdk-go&package-manager=go_modules&previous-version=0.12.0&new-version=0.13.0)](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores)

Dependabot will resolve any conflicts with this PR as long as you don't
alter it yourself. You can also trigger a rebase manually by commenting
`@dependabot rebase`.

[//]: # (dependabot-automerge-start)
[//]: # (dependabot-automerge-end)

---

<details>
<summary>Dependabot commands and options</summary>
<br />

You can trigger Dependabot actions by commenting on this PR:
- `@dependabot rebase` will rebase this PR
- `@dependabot recreate` will recreate this PR, overwriting any edits
that have been made to it
- `@dependabot merge` will merge this PR after your CI passes on it
- `@dependabot squash and merge` will squash and merge this PR after
your CI passes on it
- `@dependabot cancel merge` will cancel a previously requested merge
and block automerging
- `@dependabot reopen` will reopen this PR if it is closed
- `@dependabot close` will close this PR and stop Dependabot recreating
it. You can achieve the same result by closing it manually
- `@dependabot ignore this major version` will close this PR and stop
Dependabot creating any more for this major version (unless you reopen
the PR or upgrade to it yourself)
- `@dependabot ignore this minor version` will close this PR and stop
Dependabot creating any more for this minor version (unless you reopen
the PR or upgrade to it yourself)
- `@dependabot ignore this dependency` will close this PR and stop
Dependabot creating any more for this dependency (unless you reopen the
PR or upgrade to it yourself)


</details>

---------

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Serge Smertin <serge.smertin@databricks.com>
2023-07-18 15:30:00 +00:00
Fabian Jakobs 8cfb1c133e
First look for databricks.yml before falling back to bundle.yml (#580)
## Changes
* Add support for using `databricks.yml` as config file. If
`databricks.yml` is not found then falling back to `bundle.yml` for
backwards compatibility.
* Add support for `.yaml` extension.
* Give an error when more than one config file is found

## Tests
* added unit test
* manual testing the different cases

---------

Co-authored-by: Pieter Noordhuis <pieter.noordhuis@databricks.com>
2023-07-18 12:16:34 +02:00
shreyas-goenka a7a109a5d8
Remove base path checks during sync (#576)
## Changes
Earlier we removed recursive deletion from sync. This makes it safe
enough for us to not restrict sync to just the namespace of the user.

This PR removes that base path validation. 

Note: If the sync destination is under `/Repos` we still only create
missing directories required if the path is under my namespace ie
matches `/Repos/@me/`

## Tests
Manually

Before:
```
shreyas.goenka@THW32HFW6T hello-bundle % cli bundle deploy
Starting upload of bundle files
Error: path must be nested under /Users/shreyas.goenka@databricks.com or /Repos/shreyas.goenka@databricks.com
```

After:
```
shreyas.goenka@THW32HFW6T hello-bundle % cli bundle deploy
Starting upload of bundle files
Uploaded bundle files at /Shared/common-test/hello-bundle/files!

Starting resource deployment
Resource deployment completed!
```
2023-07-14 11:43:20 +02:00
Miles Yucht 9a0888126c
Improve auth login experience (#570)
## Changes
Currently, `databricks auth login` is difficult to use. If a user types
this command in, the command fails with
```
Error: init: cannot fetch credentials
```
after prompting for a profile name.

To make this experience smoother, this change ensures that the host, and
if necessary, the account ID, are prompted for input from the user if
they aren't provided on the CLI.


## Tests
Manual tests:

```
$ ./cli auth token                   
Databricks Host: https://<HOST>.staging.cloud.databricks.com
{
  "access_token": "...",
  "token_type": "Bearer",
  "expiry": "2023-07-11T12:56:59.929671+02:00"
}
$ ./cli auth login
Databricks Host: https://<HOST>.staging.cloud.databricks.com
Databricks Profile Name: <HOST>-test
Profile <HOST>-test was successfully saved
$ ./cli auth login
Databricks Host: https://accounts.cloud.databricks.com
Databricks Account ID: <ACCOUNTID>
Databricks Profile Name: ACCOUNT-<ACCOUNTID>-test
Profile ACCOUNT-<ACCOUNTID>-test was successfully saved 
```

---------

Co-authored-by: Pieter Noordhuis <pieter.noordhuis@databricks.com>
2023-07-12 15:36:09 +00:00
shreyas-goenka f00488d81d
Disallow notebooks in paths where files are expected (#573)
## Changes
Uploading a notebook strips it's file extension. This PR returns an
error if a notebook is specified where a file is expected. For example:
A spark python task in a job or `libraries.file.path` DLT library (where
instead `libraries.notebook.path` should be used

This PR also adds test coverage for the opposite case, when files are
not notebooks where notebooks are expected.

## Tests
Integration tests and manually
2023-07-12 12:25:00 +00:00
Andrew Nester 650fb0e8b6
Correctly use --profile flag passed for all bundle commands (#571)
## Changes
Correctly use --profile flag passed for all bundle commands.

Also adds a validation that if bundle configured host mismatches
provided profile, it throws an error.

Co-authored-by: Pieter Noordhuis <pieter.noordhuis@databricks.com>
2023-07-12 14:09:25 +02:00
dependabot[bot] 14cfc80666
Bump golang.org/x/mod from 0.11.0 to 0.12.0 (#568) 2023-07-12 11:49:35 +00:00
dependabot[bot] d28133fd3e
Bump golang.org/x/oauth2 from 0.9.0 to 0.10.0 (#566) 2023-07-12 11:42:46 +00:00
dependabot[bot] d2b7c6d611
Bump golang.org/x/term from 0.9.0 to 0.10.0 (#567) 2023-07-12 11:36:07 +00:00
Miles Yucht f203731fe6
Support tab completion for profiles (#572)
## Changes
Currently, `databricks --profile <TAB>` autocompletes with the shell
default behavior, listing files in the local directory. This is not a
great experience. Especially given that the suggested profile names for
accounts are so long, it can be cumbersome to type them out by hand.
This PR configures autocompletion for `--profile` to inspect the
profiles of ~/.databrickscfg.

One potential improvement is to filter the response based on whether the
command is known to be account-level or workspace-level.

## Tests
Manual test.
<img width="579" alt="Screenshot_11_07_2023__18_31"
src="https://github.com/databricks/cli/assets/1850319/d7a3acd0-2511-45ac-bd82-95567775c10a">
2023-07-12 12:05:51 +02:00
Lennart Kats (databricks) 57e75d3e22
Add development runs (#522)
This implements the "development run" functionality that we desire for DABs in the workspace / IDE.

## bundle.yml changes

In bundle.yml, there should be a "dev" environment that is marked as
`mode: debug`:
```
environments:
  dev:
    default: true
    mode: development # future accepted values might include pull_request, production
```

Setting `mode` to `development` indicates that this environment is used
just for running things for development. This results in several changes
to deployed assets:
* All assets will get '[dev]' in their name and will get a 'dev' tag
* All assets will be hidden from the list of assets (future work; e.g.
for jobs we would have a special job_type that hides it from the list)
* All deployed assets will be ephemeral (future work, we need some form
of garbage collection)
* Pipelines will be marked as 'development: true'
* Jobs can run on development compute through the `--compute` parameter
in the CLI
* Jobs get their schedule / triggers paused
* Jobs get concurrent runs (it's really annoying if your runs get
skipped because the last run was still in progress)

Other accepted values for `mode` are `default` (which does nothing) and
`pull-request` (which is reserved for future use).

## CLI changes

To run a single job called "shark_sighting" on existing compute, use the
following commands:
```
$ databricks bundle deploy --compute 0617-201942-9yd9g8ix
$ databricks bundle run shark_sighting
```

which would deploy and run a job called "[dev] shark_sightings" on the
compute provided. Note that `--compute` is not accepted in production
environments, so we show an error if `mode: development` is not used.

The `run --deploy` command offers a convenient shorthand for the common
combination of deploying & running:
```
$ export DATABRICKS_COMPUTE=0617-201942-9yd9g8ix
$ bundle run --deploy shark_sightings
```
The `--deploy` addition isn't really essential and I welcome feedback 🤔
I played with the idea of a "debug" or "dev" command but that seemed to
only make the option space even broader for users. The above could work
well with an IDE or workspace that automatically sets the target
compute.

One more thing I added is`run --no-wait` can now be used to run
something without waiting for it to be completed (useful for IDE-like
environments that can display progress themselves).
```
$ bundle run --deploy shark_sightings --no-wait
```
2023-07-12 08:51:54 +02:00
Pieter Noordhuis e11704618c
Release v0.200.2 (#564)
## Changes

CLI:
* Fix secrets put-secret command
([#545](https://github.com/databricks/cli/pull/545)).
* Fixed ignoring required positional parameters when --json flag is
provided ([#535](https://github.com/databricks/cli/pull/535)).
* Update cp help message to not require file scheme
([#554](https://github.com/databricks/cli/pull/554)).

Bundles:
* Fix: bundle destroy fails when bundle.tf.json file is deleted
([#519](https://github.com/databricks/cli/pull/519)).
* Fixed error reporting when included invalid files in include section
([#543](https://github.com/databricks/cli/pull/543)).
* Make top level workspace optional in JSON schema
([#562](https://github.com/databricks/cli/pull/562)).
* Propagate TF_CLI_CONFIG_FILE env variable
([#555](https://github.com/databricks/cli/pull/555)).
* Update Terraform provider schema structs
([#563](https://github.com/databricks/cli/pull/563)).
* Update inline JSON schema documentation
([#557](https://github.com/databricks/cli/pull/557)).

Dependencies:
* Bump Go SDK to v0.12.0
([#540](https://github.com/databricks/cli/pull/540)).
* Bump github.com/hashicorp/terraform-json from 0.17.0 to 0.17.1
([#541](https://github.com/databricks/cli/pull/541)).
2023-07-10 13:26:19 +02:00