Commit Graph

1379 Commits

Author SHA1 Message Date
stikkireddy 533234f148
Fix: bundle destroy fails when bundle.tf.json file is deleted (#519)
## Changes

Adds the following steps to the destroy phase:
1. interpolate
2. write

Resolves #518 

## Tests

Tested manually due there not being an examples for tests to use.
2023-07-05 21:58:06 +02:00
Pieter Noordhuis db6313e99c
Fix secrets put-secret command (#545)
## Changes

Two issues with this command:
* The command line arguments for the secret value were ignored
* If the secret value was piped through stdin, it would still prompt

The second issue prevented users from using multi-line strings because
the prompt reads until end-of-line.

This change adds testing infrastructure for:
* Setting up a workspace focused test (common between many tests)
* Running a snippet of Python through the command execution API

Porting more integration tests to use this infrastructure will be done
in later commits.

## Tests

New integration test passes.

The interactive path cannot be integration tested just yet.
2023-07-05 17:30:54 +02:00
dependabot[bot] 335475095e
Bump github.com/hashicorp/terraform-json from 0.17.0 to 0.17.1 (#541) 2023-07-03 12:03:40 +00:00
Andrew Nester 17ed7a317b
Fixed ignoring required positional parameters when --json flag is provided (#535)
## Changes
When there are positional required parameters in the command which can't
be unmarshalled from JSON, we should require them despite the fact
`--json` flag is provided.

The reason is that for some of the command, for example, `databricks
groups patch ID` these arguments are actually path arguments in API and
can't be set as part of `--json` body provided.

Original change which introduced this ignore logic is here:
https://github.com/databricks/cli/pull/405

Fixes https://github.com/databricks/cli/issues/533,
https://github.com/databricks/cli/issues/537

Note: Code generation is based on the change in this PR:
https://github.com/databricks/databricks-sdk-go/pull/536

## Tests
1. Running `cli groups patch 123 --json {...}` works correctly

Backward compatibility tests with previous changes from
https://github.com/databricks/cli/pull/405
1. `cli clusters events --json '{"cluster_id": "1029-xxxx"}'` - works,
returns list of events
2. `cli clusters events 1029-xxxx` - works, returns list of events
3. `cli clusters events` - works, first prompts for Cluster ID and then
returns the list of events
2023-07-03 13:20:30 +02:00
Pieter Noordhuis ad8183d7a9
Bump Go SDK to v0.12.0 (#540)
## Changes

* Regenerate CLI commands
* Ignore `account-access-control-proxy` (see #505)

## Tests

Unit and integration tests pass.
2023-07-03 11:46:45 +02:00
Pieter Noordhuis f42279fe47
Release v0.200.1 (#532)
## Changes

CLI:
* Add --absolute flag for ls command
([#508](https://github.com/databricks/cli/pull/508)).
* Add dbfs scheme prefix to paths in cp command output
([#516](https://github.com/databricks/cli/pull/516)).
* Add provider detection to the repos create command
([#528](https://github.com/databricks/cli/pull/528)).
* Added configure-cluster flag for auth login
([#500](https://github.com/databricks/cli/pull/500)).
* Added prompts for Databricks profile for auth login command
([#502](https://github.com/databricks/cli/pull/502)).
* Allow specifying repo by path for repos commands
([#526](https://github.com/databricks/cli/pull/526)).
* Decode contents by default in workspace export command
([#531](https://github.com/databricks/cli/pull/531)).
* Fixed jobs create command to only accept JSON payload
([#498](https://github.com/databricks/cli/pull/498)).
* Make local files default for fs commands
([#506](https://github.com/databricks/cli/pull/506)).
* Remove \r from new line print statments
([#509](https://github.com/databricks/cli/pull/509)).
* Remove extra call to filer.Stat in dbfs filer.Read
([#515](https://github.com/databricks/cli/pull/515)).
* Update alerts command integration test
([#512](https://github.com/databricks/cli/pull/512)).
* Update variable regex to support hyphens
([#503](https://github.com/databricks/cli/pull/503)).

Bundles:
* Add DATABRICKS_BUNDLE_TMP env variable
([#462](https://github.com/databricks/cli/pull/462)).
* Update Terraform provider schema structs
([#504](https://github.com/databricks/cli/pull/504)).

Dependencies:
* Bump github.com/databricks/databricks-sdk-go from
0.9.1-0.20230614092458-b5bbc1c8dabb to 0.10.0
([#497](https://github.com/databricks/cli/pull/497)).

Internal:
* Use direct download for workspace filer read
([#514](https://github.com/databricks/cli/pull/514)).
2023-06-28 14:23:55 +00:00
Pieter Noordhuis f64c44285f
Decode contents by default in workspace export command (#531)
## Changes

Also see #525.

The direct download flag has been removed in newer versions because of
the content type issue.

Instead, we can make the command decode the base64 output when the
output mode is text.

```
$ databricks workspace export /some/path/script.sh
#!/bin/bash
echo "this is a script"
```

## Tests

New integration test.
2023-06-27 20:42:29 +02:00
shreyas-goenka 54148ffedf
Add --absolute flag for ls command (#508)
tested manually

---------

Co-authored-by: Pieter Noordhuis <pieter.noordhuis@databricks.com>
2023-06-27 16:06:56 +02:00
shreyas-goenka 1f5cdfda29
Add dbfs scheme prefix to paths in cp command output (#516)
## Changes
Adds the dbfs:/ prefix to paths output by the cp command so they can be
used with the CLI

## Tests
Manually

Currently there are no integration tests for command output, I'll add
them in separately

---------

Co-authored-by: Pieter Noordhuis <pieter.noordhuis@databricks.com>
2023-06-27 14:42:27 +02:00
Pieter Noordhuis c64a3336c5
Add provider detection to the repos create command (#528)
## Changes

Closes #527.

## Tests

Integration tests pass.
2023-06-27 14:17:13 +02:00
Pieter Noordhuis 64fcd3d2ee
Allow specifying repo by path for repos commands (#526)
## Changes

The repos commands take a repo ID. It is more convenient to specify
repos by their paths.

Closes #523.

## Tests

New integration tests pass.
2023-06-26 23:37:05 +00:00
Pieter Noordhuis cf92698eb3
Application -> Asset (#529) 2023-06-27 01:31:20 +02:00
shreyas-goenka 30efe91c6d
Make local files default for fs commands (#506)
## Changes
<!-- Summary of your changes that are easy to understand -->

## Tests
<!-- How is this tested? -->
2023-06-23 16:07:09 +02:00
shreyas-goenka d0e9953ad9
Use direct download for workspace filer read (#514)
## Changes
Use the download method from the SDK in the read method for the WSFS
implementation of the filer interface.

Closes #452.

## Tests
Tested by existing integration tests
2023-06-23 15:17:39 +02:00
shreyas-goenka f9260125aa
Remove extra call to filer.Stat in dbfs filer.Read (#515)
## Changes
This PR removes the stat call and instead relies on errors returned by
the go SDK to return the appropriate errors

## Tests
Tested using existing filer integration tests
2023-06-23 15:08:22 +02:00
stikkireddy 3c1e69a064
Update variable regex to support hyphens (#503)
## Changes

Modified interpolation logic to use:
`\$\{([a-zA-Z]+([-_]*[a-zA-Z0-9]+)*(\.[a-zA-Z]+([-_]*[a-zA-Z0-9]+)*)*)\}`

**Edit**: Suggested by @pietern
`\$\{([a-zA-Z]+([-_]?[a-zA-Z0-9]+)*(\.[a-zA-Z]+([-_]?[a-zA-Z0-9]+)*)*)\}`
to be more selective and not allow consequent hyphens or underscores to
make the keys more readable.

Explanation:
1. All interpolation starts with `${` and ends with `}`
2. All interpolated locations are split by by `.`
3. All sections are expected to start with a alphabet `[a-zA-Z]`; no
numbers, hyphens or underscores.
4. All sections are expected to end with an alphanumeric `[a-zA-Z0-9]`
no hyphens or underscores

This change allows the current interpolation to be more permissive.

**Note** it does break backwards compatibility because `[a-zA-Z] !=
[\w]`. `\w` includes alphanumeric and underscores. `\w = [a-zA-Z0-9_]`

## Tests
There are two tests with examples of valid and invalid interpolation and
a test to validate expansion.
2023-06-23 12:56:54 +02:00
Pieter Noordhuis ae13135fc6
Follow up to #513 (#521)
This reverts changes from #513 which ended up not being necessary.
2023-06-23 12:51:31 +02:00
Andrew Nester d23d4aef3a
Fixed error on multiple profiles and failure to create a new profile with configured cluster (#513) 2023-06-22 17:20:10 +02:00
shreyas-goenka f2a2d058d1
Remove \r from new line print statments (#509)
## Changes
Removes carriage character from new line prints for json output mode and
sync events

## Tests
Manually
2023-06-22 13:47:52 +02:00
Pieter Noordhuis 5c23c430be
Update alerts command integration test (#512) 2023-06-22 10:30:52 +02:00
Pawarit Laosunthara 5ed100ea37
Fix typo in README.md (#510) 2023-06-21 20:26:09 +02:00
Andrew Nester b911327d62
Added configure-cluster flag for auth login (#500)
## Changes
Added configure-cluster flag for auth login which will allow to
configure cluster ID and save it in Databricks profile

Note: the build will fail until this one is merged and released
https://github.com/databricks/databricks-sdk-go/pull/524

## Tests

```
andrew.nester@HFW9Y94129 cli % ./cli auth login https://xxxxxxx.databricks.com --configure-cluster
✔ Databricks Profile Name: my-profile█
Search: █
? Choose cluster:
  10.1 ML beta (1029-yyyyy-xxxxxx)
  10.5 ML standard cluster
  12.2 LTS
↓ 13.1 free for all


andrew.nester@HFW9Y94129 cli % cat ~/.databrickscfg
[DEFAULT]
host       = https://xxxxx.databricks.com
cluster_id = 1029-xxxxx-yyyyy
auth_type  = databricks-cli
```
2023-06-21 17:51:59 +02:00
Andrew Nester 7db1990c56
Added prompts for Databricks profile for auth login command (#502)
## Changes
Added prompts for Databricks profile for auth login command

## Tests
```
andrew.nester@HFW9Y94129 cli % ./cli auth login https://xxxx-databricks.com
✔ Databricks Profile Name: my-profile█
```
2023-06-21 12:58:28 +02:00
Gleb Kanterov ccbcccd929
Add DATABRICKS_BUNDLE_TMP env variable (#462)
## Changes
Add DATABRICKS_BUNDLE_TMP env variable. It allows using a temporary
directory instead of writing to `$CWD/.databricks/bundle`

## Tests
I added unit tests

---------

Co-authored-by: Pieter Noordhuis <pieter.noordhuis@databricks.com>
2023-06-21 09:53:54 +02:00
dependabot[bot] 7fb34e4767
Bump github.com/databricks/databricks-sdk-go from 0.9.1-0.20230614092458-b5bbc1c8dabb to 0.10.0 (#497) 2023-06-21 07:43:07 +00:00
Andrew Nester d11128c638
Fixed jobs create command to only accept JSON payload (#498)
## Changes
Fixed jobs create command to only accept JSON payload.

Note: relies on this PR from Go SDK
https://github.com/databricks/databricks-sdk-go/pull/522

## Tests

```
andrew.nester@HFW9Y94129 cli % ./cli jobs create -h
Create a new job.

  Create a new job.

Usage:
  databricks jobs create [flags]

Flags:
  -h, --help        help for create
      --json JSON   either inline JSON string or @path/to/file.json with request body (default JSON (0 bytes))

Global Flags:
  -e, --environment string       bundle environment to use (if applicable)
      --log-file file            file to write logs to (default stderr)
      --log-format type          log output format (text or json) (default text)
      --log-level format         log level (default disabled)
  -o, --output type              output type: text or json (default text)
  -p, --profile string           ~/.databrickscfg profile
      --progress-format format   format for progress logs (append, inplace, json) (default default)
 ```
2023-06-21 09:25:29 +02:00
stikkireddy ddeb7487b3
Update Terraform provider schema structs (#504)
## Changes

Generated from provider version 1.19.0.

## Tests

Ran the existing unit tests and they seemed to have checked out.
2023-06-21 08:40:11 +02:00
Pieter Noordhuis 63e80a0b71
Release v0.200.0 (#499)
## Changes

This version marks the first version available as public preview.

The minor bump to 200 better disambiguates between Databricks CLI "v1" (the Python version)
and this version, Databricks CLI "v2". The minor version of 0.100 may look lower than 0.17
to some, whereas 200 does not. This bump has no other significance.

CLI:
* Add filer.Filer implementation backed by the Files API ([#474](https://github.com/databricks/cli/pull/474)).
* Add fs cp command ([#463](https://github.com/databricks/cli/pull/463)).
* Correctly set ExactArgs if generated command has positional arguments ([#488](https://github.com/databricks/cli/pull/488)).
* Do not use white color as string output ([#489](https://github.com/databricks/cli/pull/489)).
* Update README to reflect public preview status ([#491](https://github.com/databricks/cli/pull/491)).

Bundles:
* Fix force flag not working for bundle destroy ([#434](https://github.com/databricks/cli/pull/434)).
* Fix locker unlock for destroy ([#492](https://github.com/databricks/cli/pull/492)).
* Use better error assertions and clean up locker API ([#490](https://github.com/databricks/cli/pull/490)).

Dependencies:
* Bump golang.org/x/mod from 0.10.0 to 0.11.0 ([#496](https://github.com/databricks/cli/pull/496)).
* Bump golang.org/x/sync from 0.2.0 to 0.3.0 ([#495](https://github.com/databricks/cli/pull/495)).
2023-06-20 10:22:50 +02:00
dependabot[bot] d671516b39
Bump golang.org/x/mod from 0.10.0 to 0.11.0 (#496) 2023-06-20 06:37:55 +00:00
dependabot[bot] 8804b9d8ba
Bump golang.org/x/sync from 0.2.0 to 0.3.0 (#495) 2023-06-20 06:26:41 +00:00
Pieter Noordhuis e19eaca4d1
Add filer.Filer implementation backed by the Files API (#474)
## Tests

New integration test for the read/write parts of the other filers. The
integration test cannot be shared just yet because the Files API doesn't
include support for creating/listing/removing directories yet.
2023-06-19 18:29:13 +00:00
shreyas-goenka 5d036ab6b8
Fix locker unlock for destroy (#492)
## Changes
Adds ability for allowing unlock to succeed even if the deploy file is
missing.
 
## Tests
Using integration tests and manually
2023-06-19 15:57:25 +02:00
shreyas-goenka 4a03265dc2
Fix force flag not working for bundle destroy (#434)
## Changes
`--force` flag did not exist for `bundle destroy`. This PR adds that in.

## Tests
manually tested. Now adding the `--force` flag hijacks the deploy lock
on the target directory.
2023-06-19 12:31:07 +02:00
shreyas-goenka bb32067a80
Add fs cp command (#463)
## Tests
Tested using integration tests
2023-06-16 17:09:08 +02:00
Pieter Noordhuis b72742a129
Update README to reflect public preview status (#491) 2023-06-16 14:37:10 +00:00
shreyas-goenka de47cf19f1
Use better error assertions and clean up locker API (#490)
## Changes
Some cleanup work

## Tests
Locker integration test passes
2023-06-16 16:29:04 +02:00
Andrew Nester fb25baf100
Do not use white color as string output (#489)
## Changes
Do not use white color as string output
2023-06-16 13:55:22 +00:00
Andrew Nester 9cf0e0db24
Correctly set ExactArgs if generated command has positional arguments (#488)
## Changes
Some of the command such as `databricks alerts create` require
positional arguments which are not primitive.

Since these arguments are required, we should correctly set ExactArgs
for such commands

Fixes #367 

## Tests
Running `databricks alerts create`

Before
```
andrew.nester@HFW9Y94129 cli % ./cli alerts create                     
panic: runtime error: index out of range [0] with length 0

goroutine 1 [running]:
github.com/databricks/bricks/cmd/workspace/alerts.glob..func1(0x22a1280?, {0x2321638, 0x0, 0x0?})
	github.com/databricks/bricks/cmd/workspace/alerts/alerts.go:57 +0x355
github.com/spf13/cobra.(*Command).execute(0x22a1280, {0x2321638, 0x0, 0x0})
	github.com/spf13/cobra@v1.7.0/command.go:940 +0x862
github.com/spf13/cobra.(*Command).ExecuteC(0x22a0700)
	github.com/spf13/cobra@v1.7.0/command.go:1068 +0x3bd
github.com/spf13/cobra.(*Command).ExecuteContextC(...)
	github.com/spf13/cobra@v1.7.0/command.go:1001
github.com/databricks/bricks/cmd/root.Execute()
	github.com/databricks/bricks/cmd/root/root.go:80 +0x6a
main.main()
	github.com/databricks/bricks/main.go:18 +0x17                                           

```

After
```
andrew.nester@HFW9Y94129 cli % ./cli alerts create                                                                
Error: provide command input in JSON format by specifying --json option
```
Acceptance test
```
=== RUN   TestAccAlertsCreateErrWhenNoArguments
    alerts_test.go:10: gcp
    helpers.go:147: Error running command: provide command input in JSON format by specifying --json option
--- PASS: TestAccAlertsCreateErrWhenNoArguments (1.99s)
PASS
```
2023-06-16 12:15:25 +00:00
Pieter Noordhuis 18e1d93218
Release v0.100.4 (#486)
## Changes

CLI:
* Add workspace import-dir command
([#456](https://github.com/databricks/cli/pull/456)).
* Annotate generated commands with OpenAPI package name
([#466](https://github.com/databricks/cli/pull/466)).
* Associate generated commands with command groups
([#475](https://github.com/databricks/cli/pull/475)).
* Disable shell completions for generated commands
([#483](https://github.com/databricks/cli/pull/483)).
* Include [DEFAULT] section header when writing ~/.databrickscfg
([#464](https://github.com/databricks/cli/pull/464)).
* Pass through proxy related environment variables
([#465](https://github.com/databricks/cli/pull/465)).
* Restore flags to original values on test completion
([#470](https://github.com/databricks/cli/pull/470)).
* Update configure command
([#482](https://github.com/databricks/cli/pull/482)).

Dependencies:
* Bump SDK to latest
([#473](https://github.com/databricks/cli/pull/473)).
2023-06-15 15:04:11 +00:00
Pieter Noordhuis dd92b3f989
Disable shell completions for generated commands (#483)
## Changes

Disable shell completions for generated commands.

Default completion behavior completes local files which never makes
sense.

Automatic contextual completion of required arguments would be super
powerful but a lot of work to get right. Until then, we could do manual
completion functions in `overrides.go` as needed.

This fixes #374.

## Tests

Confirmed manually that commands no longer complete local files.
2023-06-15 14:56:36 +00:00
Pieter Noordhuis c080fc67c9
Associate generated commands with command groups (#475)
## Changes

With this change related commands show up next to each other in help
output.

The ordered list of groups is hard-coded until it can be derived from
the specification.

## Tests

Manually confirmed that the help output of the root command and the
account command list commands by their groups.
2023-06-15 14:47:24 +00:00
Pieter Noordhuis b9406efd27
Update configure command (#482)
## Changes

This now uses:
* libs/cmdio to determine interactivity and perform prompting
* libs/databrickscfg to persist the profile

It loads a config.Config structure from the environment just like we do
for unified authentication. It is therefore possible to specify both the
host and token with environment variables.

## Tests

```
pieter.noordhuis@L4GHXDT29P /tmp % export DATABRICKS_CONFIG_FILE=.databrickscfg
pieter.noordhuis@L4GHXDT29P /tmp % databricks configure
Databricks Host: https://foo.bar
Personal Access Token: *****
pieter.noordhuis@L4GHXDT29P /tmp % cat .databrickscfg
[DEFAULT]
host  = https://foo.bar
token = token
pieter.noordhuis@L4GHXDT29P /tmp % echo token | databricks configure
Error: host must be set in non-interactive mode
pieter.noordhuis@L4GHXDT29P /tmp % echo token | databricks configure --host foo
Error: must start with https://
pieter.noordhuis@L4GHXDT29P /tmp % echo token | databricks configure --host https://foo
pieter.noordhuis@L4GHXDT29P /tmp % cat .databrickscfg
[DEFAULT]
host  = https://foo
token = token
pieter.noordhuis@L4GHXDT29P /tmp % cat .databrickscfg
pieter.noordhuis@L4GHXDT29P /tmp % databricks configure --host https://foo
Personal Access Token: ******
pieter.noordhuis@L4GHXDT29P /tmp % cat .databrickscfg
[DEFAULT]
host  = https://foo
token = token2
```
2023-06-15 12:50:19 +00:00
Pieter Noordhuis becf4d9c31
Fix deprecation notice for io/ioutil (#477)
## Changes

"io/ioutil" has been deprecated since Go 1.16: As of Go 1.16, the same
functionality is now provided by package io or package os, and those
implementations should be preferred in new code. See the specific
function documentation for details.

## Tests

n/a
2023-06-15 10:38:30 +00:00
Pieter Noordhuis faeddfae68
Update short name for bundle command (#476) 2023-06-14 22:44:18 +02:00
Pieter Noordhuis 1875908b59
Pass through proxy related environment variables (#465)
## Changes

If set on the host, we must pass them through to Terraform.

## Tests

Unit tests pass.
2023-06-14 21:58:26 +02:00
Pieter Noordhuis 30e9cf048c
Bump SDK to latest (#473)
## Changes

This pulls in the Files API work from
https://github.com/databricks/databricks-sdk-go/pull/423.

## Tests

Integration tests pass.
2023-06-14 18:27:29 +00:00
Serge Smertin 6b1b3dfa87
Create NOTICE (#468) 2023-06-14 20:05:08 +02:00
Pieter Noordhuis 96643fd746
Hide commands under preview from help output (#469)
## Changes

TSIA

Depends on https://github.com/databricks/databricks-sdk-go/pull/497.

## Tests

Manually looked at the help output and no longer see the hidden commands.
2023-06-14 14:59:57 +00:00
Serge Smertin 63437a6a98
Create LICENSE (#467)
Adding Databricks License v2
2023-06-14 14:38:52 +00:00
shreyas-goenka 968489d7c4
Restore flags to original values on test completion (#470)
## Changes

This is necessary to avoid test interference.

## Tests

Manually.

---------

Co-authored-by: Pieter Noordhuis <pieter.noordhuis@databricks.com>
2023-06-14 14:53:27 +02:00