Commit Graph

489 Commits

Author SHA1 Message Date
Pieter Noordhuis c080fc67c9
Associate generated commands with command groups (#475)
## Changes

With this change related commands show up next to each other in help
output.

The ordered list of groups is hard-coded until it can be derived from
the specification.

## Tests

Manually confirmed that the help output of the root command and the
account command list commands by their groups.
2023-06-15 14:47:24 +00:00
Pieter Noordhuis b9406efd27
Update configure command (#482)
## Changes

This now uses:
* libs/cmdio to determine interactivity and perform prompting
* libs/databrickscfg to persist the profile

It loads a config.Config structure from the environment just like we do
for unified authentication. It is therefore possible to specify both the
host and token with environment variables.

## Tests

```
pieter.noordhuis@L4GHXDT29P /tmp % export DATABRICKS_CONFIG_FILE=.databrickscfg
pieter.noordhuis@L4GHXDT29P /tmp % databricks configure
Databricks Host: https://foo.bar
Personal Access Token: *****
pieter.noordhuis@L4GHXDT29P /tmp % cat .databrickscfg
[DEFAULT]
host  = https://foo.bar
token = token
pieter.noordhuis@L4GHXDT29P /tmp % echo token | databricks configure
Error: host must be set in non-interactive mode
pieter.noordhuis@L4GHXDT29P /tmp % echo token | databricks configure --host foo
Error: must start with https://
pieter.noordhuis@L4GHXDT29P /tmp % echo token | databricks configure --host https://foo
pieter.noordhuis@L4GHXDT29P /tmp % cat .databrickscfg
[DEFAULT]
host  = https://foo
token = token
pieter.noordhuis@L4GHXDT29P /tmp % cat .databrickscfg
pieter.noordhuis@L4GHXDT29P /tmp % databricks configure --host https://foo
Personal Access Token: ******
pieter.noordhuis@L4GHXDT29P /tmp % cat .databrickscfg
[DEFAULT]
host  = https://foo
token = token2
```
2023-06-15 12:50:19 +00:00
Pieter Noordhuis becf4d9c31
Fix deprecation notice for io/ioutil (#477)
## Changes

"io/ioutil" has been deprecated since Go 1.16: As of Go 1.16, the same
functionality is now provided by package io or package os, and those
implementations should be preferred in new code. See the specific
function documentation for details.

## Tests

n/a
2023-06-15 10:38:30 +00:00
Pieter Noordhuis faeddfae68
Update short name for bundle command (#476) 2023-06-14 22:44:18 +02:00
Pieter Noordhuis 1875908b59
Pass through proxy related environment variables (#465)
## Changes

If set on the host, we must pass them through to Terraform.

## Tests

Unit tests pass.
2023-06-14 21:58:26 +02:00
Pieter Noordhuis 30e9cf048c
Bump SDK to latest (#473)
## Changes

This pulls in the Files API work from
https://github.com/databricks/databricks-sdk-go/pull/423.

## Tests

Integration tests pass.
2023-06-14 18:27:29 +00:00
Serge Smertin 6b1b3dfa87
Create NOTICE (#468) 2023-06-14 20:05:08 +02:00
Pieter Noordhuis 96643fd746
Hide commands under preview from help output (#469)
## Changes

TSIA

Depends on https://github.com/databricks/databricks-sdk-go/pull/497.

## Tests

Manually looked at the help output and no longer see the hidden commands.
2023-06-14 14:59:57 +00:00
Serge Smertin 63437a6a98
Create LICENSE (#467)
Adding Databricks License v2
2023-06-14 14:38:52 +00:00
shreyas-goenka 968489d7c4
Restore flags to original values on test completion (#470)
## Changes

This is necessary to avoid test interference.

## Tests

Manually.

---------

Co-authored-by: Pieter Noordhuis <pieter.noordhuis@databricks.com>
2023-06-14 14:53:27 +02:00
shreyas-goenka 3d03df8844
Add mode default as a valid set value for progress-format flag (#472)
Manually tested `progress-format` works fine for all three modes
2023-06-14 13:26:56 +02:00
Pieter Noordhuis a17876480a
Include [DEFAULT] section header when writing ~/.databrickscfg (#464)
## Changes

The ini library omits the default section header and in doing so breaks
compatibility with Python's config parser. It raises:

```
Error: MissingSectionHeaderError: File contains no section headers.
```

This commit makes sure the DEFAULT section header is included.

If the config file doesn't include a DEFAULT section itself, we include
a comment describing its purpose.

## Tests

New tests pass. Manually confirmed the DEFAULT section header is
included.

---------

Co-authored-by: PaulCornellDB <paul.cornell@databricks.com>
2023-06-13 16:41:56 +00:00
Pieter Noordhuis f219a0da5a
Annotate generated commands with OpenAPI package name (#466)
Co-authored-by: Serge Smertin <259697+nfx@users.noreply.github.com>
2023-06-13 17:20:42 +02:00
shreyas-goenka d38649088c
Add workspace import-dir command (#456)
## Tests
Testing using integration tests and manually
2023-06-12 21:03:46 +02:00
Pieter Noordhuis 35dd56c716
Release v0.100.3 (#461)
## Changes

CLI:
* Add directory tracking to sync
([#425](https://github.com/databricks/cli/pull/425)).
* Add fs cat command for dbfs files
([#430](https://github.com/databricks/cli/pull/430)).
* Add fs ls command for dbfs
([#429](https://github.com/databricks/cli/pull/429)).
* Add fs mkdirs command for dbfs
([#432](https://github.com/databricks/cli/pull/432)).
* Add fs rm command for dbfs
([#433](https://github.com/databricks/cli/pull/433)).
* Add installation instructions
([#458](https://github.com/databricks/cli/pull/458)).
* Add new line to cmdio JSON rendering
([#443](https://github.com/databricks/cli/pull/443)).
* Add profile on `databricks auth login`
([#423](https://github.com/databricks/cli/pull/423)).
* Add readable console logger
([#370](https://github.com/databricks/cli/pull/370)).
* Add workspace export-dir command
([#449](https://github.com/databricks/cli/pull/449)).
* Added secrets input prompt for secrets put-secret command
([#413](https://github.com/databricks/cli/pull/413)).
* Added spinner when loading command prompts
([#420](https://github.com/databricks/cli/pull/420)).
* Better error message if can not load prompts
([#437](https://github.com/databricks/cli/pull/437)).
* Changed service template to correctly handle required positional
arguments ([#405](https://github.com/databricks/cli/pull/405)).
* Do not generate prompts for certain commands
([#438](https://github.com/databricks/cli/pull/438)).
* Do not prompt for List methods
([#411](https://github.com/databricks/cli/pull/411)).
* Do not use FgWhite and FgBlack for terminal output
([#435](https://github.com/databricks/cli/pull/435)).
* Skip path translation of job task for jobs with a Git source
([#404](https://github.com/databricks/cli/pull/404)).
* Tweak profile prompt
([#454](https://github.com/databricks/cli/pull/454)).
* Update with the latest Go SDK
([#457](https://github.com/databricks/cli/pull/457)).
* Use cmdio in version command for `--output` flag
([#419](https://github.com/databricks/cli/pull/419)).

Bundles:
* Check for nil environment before accessing it
([#453](https://github.com/databricks/cli/pull/453)).

Dependencies:
* Bump github.com/hashicorp/terraform-json from 0.16.0 to 0.17.0
([#459](https://github.com/databricks/cli/pull/459)).
* Bump github.com/mattn/go-isatty from 0.0.18 to 0.0.19
([#412](https://github.com/databricks/cli/pull/412)).

Internal:
* Add Mkdir and ReadDir functions to filer.Filer interface
([#414](https://github.com/databricks/cli/pull/414)).
* Add Stat function to filer.Filer interface
([#421](https://github.com/databricks/cli/pull/421)).
* Add check for path is a directory in filer.ReadDir
([#426](https://github.com/databricks/cli/pull/426)).
* Add fs.FS adapter for the filer interface
([#422](https://github.com/databricks/cli/pull/422)).
* Add implementation of filer.Filer for local filesystem
([#460](https://github.com/databricks/cli/pull/460)).
* Allow equivalence checking of filer errors to fs errors
([#416](https://github.com/databricks/cli/pull/416)).
* Fix locker integration test
([#417](https://github.com/databricks/cli/pull/417)).
* Implement DBFS filer
([#139](https://github.com/databricks/cli/pull/139)).
* Include recursive deletion in filer interface
([#442](https://github.com/databricks/cli/pull/442)).
* Make filer.Filer return fs.DirEntry from ReadDir
([#415](https://github.com/databricks/cli/pull/415)).
* Speed up sync integration tests
([#428](https://github.com/databricks/cli/pull/428)).
2023-06-12 17:08:00 +02:00
Lennart Kats (databricks) 03964537b2
Add installation instructions (#458)
## Changes

Add installation instructions to the README.
2023-06-12 16:31:22 +02:00
dependabot[bot] 2c2b8655a6
Bump github.com/hashicorp/terraform-json from 0.16.0 to 0.17.0 (#459) 2023-06-12 14:09:41 +00:00
Pieter Noordhuis 960ce2e18e
Add implementation of filer.Filer for local filesystem (#460)
## Changes

Local file reads on Windows require the file handle to be closed after
using it. This commit includes an interface change to return an
`io.ReadCloser` from `Read` to accommodate this.

## Tests

The existing integration tests for the filer interface all pass.
2023-06-12 15:53:58 +02:00
Serge Smertin 2aa61a7c1b
Update with the latest Go SDK (#457)
## Changes
- removed deprecated methods
- regenerated with the latest OpenAPI spec
- picked up the latest go SDK version

## Tests
`make test`
2023-06-12 14:23:21 +02:00
shreyas-goenka cd99fce2a5
Add fs mkdirs command for dbfs (#432)
## Changes
TSIA

## Tests
Integration tests
2023-06-12 14:05:56 +02:00
Pieter Noordhuis 16bb224108
Add directory tracking to sync (#425)
## Changes

This change replaces usage of the `repofiles` package with the `filer`
package to consolidate WSFS code paths.

The `repofiles` package implemented the following behavior. If a file at
`foo/bar.txt` was created and removed, the directory `foo` was kept
around because we do not perform directory tracking. If subsequently, a
file at `foo` was created, it resulted in an `fs.ErrExist` because it is
impossible to overwrite a directory. It would then perform a recursive
delete of the path if this happened and retry the file write.

To make this use case work without resorting to a recursive delete on
conflict, we need to implement directory tracking as part of sync. The
approach in this commit is as follows:

1. Maintain set of directories needed for current set of files. Compare
to previous set of files. This results in mkdir of added directories and
rmdir of removed directories.
2. Creation of new directories should happen prior to writing files.
Otherwise, many file writes may race to create the same parent
directories, resulting in additional API calls. Removal of existing
directories should happen after removing files.
3. Making new directories can be deduped across common prefixes where
only the longest prefix is created recursively.
4. Removing existing directories must happen sequentially, starting with
the longest prefix.
5. Removal of directories is a best effort. It fails only if the
directory is not empty, and if this happens we know something placed a
file or directory manually, outside of sync.

## Tests

* Existing integration tests pass (modified where it used to assert
directories weren't cleaned up)
* New integration test to confirm the inability to remove a directory
doesn't fail the sync run
2023-06-12 11:44:00 +00:00
Pieter Noordhuis e4415bfbcf
Tweak profile prompt (#454)
## Changes

This includes the following changes:
* Move profile loading code to libs/databrickscfg and add tests
* Update prompt label to reflect workspace/account profiles
* Start prompt in search mode by default
* Custom error if `~/.databrickscfg` doesn't exist
* Custom error if `~/.databrickscfg` doesn't contain profiles
* Use stderr for prompt so that stdout redirection works (e.g. with `jq` or `jless`)

## Tests

* New unit tests pass
* Manual tests for both workspace and account commands
* Search-by-default is really nice if you have many profiles
2023-06-09 13:56:35 +02:00
Pieter Noordhuis 894d25e434
Check for nil environment before accessing it (#453) 2023-06-08 20:55:49 +00:00
shreyas-goenka 4818541062
Add workspace export-dir command (#449)
## Changes
This PR:
1. Adds the export-dir command
2. Changes filer.Read to return an error if a user tries to read a
directory
3. Adds returning internal file structures from filer.Stat().Sys()

## Tests
Integration tests and manually
2023-06-08 18:15:12 +02:00
shreyas-goenka 53164ae880
Add new line to cmdio JSON rendering (#443)
## Changes
This PR adds a new line break to JSON rendering using cmdio. This is
useful when we call `cmdio.Render` multiple times

## Tests
Manually

Co-authored-by: Pieter Noordhuis <pieter.noordhuis@databricks.com>
2023-06-08 15:48:51 +02:00
stikkireddy 402fcdd62c
Skip path translation of job task for jobs with a Git source (#404)
## Changes

Added skipping of translating paths for notebook path in notebook tasks
and python file path in spark python tasks if the git source is not null.

Resolves: #402

## Tests

There is a unit test and also tested with a sample bundle:

```
resources:
  jobs:
    demo:
      git_source:
        git_branch: master
        git_provider: github
        git_url: https://github.com/test/dummy
   ....
```

---------

Co-authored-by: Pieter Noordhuis <pieter.noordhuis@databricks.com>
2023-06-07 12:34:59 +02:00
Pieter Noordhuis be10ff9a75
Include recursive deletion in filer interface (#442)
## Changes

This captures the recursive deletion of a directory tree in the filer interface.

Prompted by #433.

## Tests

Integration tests pass (ran the filer ones on AWS and Azure).
2023-06-06 06:27:47 +00:00
shreyas-goenka d6d35e314f
Add fs rm command for dbfs (#433)
## Changes
Please look at the title

## Tests
Integration tests
2023-06-05 23:21:47 +00:00
shreyas-goenka ae10419eb8
Add fs cat command for dbfs files (#430)
## Changes
TSIA

## Tests
Manually and integration tests
2023-06-06 01:16:23 +02:00
Andrew Nester df3f5863c7
Do not generate prompts for certain commands (#438)
## Changes
Some of the commands do not support prompts, for example `workspace
get-status` but we were wrongly suggesting customers some option.

Quick fix for this is not to provide prompts for these known commands.

Note: it uses a method from this PR in Go SDK
https://github.com/databricks/databricks-sdk-go/pull/416

## Tests
Running `workspace get-status`

Before
```
andrew.nester@HFW9Y94129 multiples-tasks % ../../cli/cli workspace get-status
Error: Path () doesn't start with '/'
```

After
```
andrew.nester@HFW9Y94129 multiples-tasks % ../../cli/cli workspace get-status
Error: accepts 1 arg(s), received 0

```
2023-06-05 19:38:45 +02:00
shreyas-goenka 6ff00122ad
Add fs ls command for dbfs (#429)
## Changes
1. Adds fs ls command
2. Adds ability to define multiple templates

## Tests
Manually and integration tests
2023-06-05 17:41:30 +02:00
Andrew Nester 1f130f3722
Do not use FgWhite and FgBlack for terminal output (#435)
## Changes
Using white / black color for terminal output will lead to poorly
displayed content in either light or dark terminal backgrounds. Some
other CLIs experienced same issues
(https://github.com/qri-io/qri/pull/774)

Instead, let's just use color to highlight some of the output so it's
more compatible with different background styles

## Tests
<img width="772" alt="Screenshot 2023-06-05 at 16 05 09"
src="https://github.com/databricks/cli/assets/2969996/01790239-6a33-4059-86a8-d5117ea0b75f">

---

<img width="757" alt="Screenshot 2023-06-05 at 16 05 20"
src="https://github.com/databricks/cli/assets/2969996/ea3b9fdc-3782-4f4f-a9df-19e66af0c04f">
2023-06-05 17:30:40 +02:00
Andrew Nester 3dbf7a575a
Better error message if can not load prompts (#437)
## Changes
Better error message if can not load prompts

## Tests
Setup 2 jobs with the same name and ran `cli job get`

```
andrew.nester@HFW9Y94129 multiples-tasks % ../../cli/cli jobs get
Error: failed to load names for Jobs drop-down. Please manually specify required arguments. Original error: duplicate .Settings.Name: duplicatejob
```
2023-06-05 16:32:03 +02:00
Pieter Noordhuis 28d36577a2
Speed up sync integration tests (#428)
## Changes

This change implements:
* Channels for line-by-line output from stdout/stderr
* A function to wait for a sync step to complete (using above)
* Ensure all tests are prefixed `TestAccSync`
* Use temporary paths in WSFS instead of cloning a repo

## Tests

The same integration tests now pass in ~90 seconds (was ~250s).
2023-06-02 14:02:18 +00:00
Pieter Noordhuis 1c0d67f66c
Add fs.FS adapter for the filer interface (#422)
## Changes

This enables the use of `io/fs` functions `fs.Glob` and `fs.WalkDir`
with filers.

We can't use `fs.FS` as the standard interface instead of `filer.Filer` because:
1. It was made for reading from filesystems only, not writing
2. It doesn't take a context for the core functions

Therefore a wrapper will do.

## Tests

* Added unit tests to cover the adapter through a fake filer.
* Manually ran `fs.WalkDir` against both WSFS and DBFS filers.
2023-06-02 12:49:59 +00:00
Andrew Nester f8255f356b
Added spinner when loading command prompts (#420)
## Changes
Added spinner when loading command prompts

## Tests
```
andrew.nester@HFW9Y94129 multiples-tasks % cli jobs get-run
⡿ No RUN_ID argument specified. Loading names for Jobs drop-down.```
2023-06-02 14:13:18 +02:00
Serge Smertin a6c9533c1c
Add profile on `databricks auth login` (#423)
## Changes
- added saving profile to `~/.databrickscfg` whenever we do `databricks
auth login`.
- we either match profile by account id / canonical host or introduce
the new one from deployment name.
- fail on multiple profiles with matching accounts or workspace hosts.
- overriding `~/.databrickscfg` keeps the (valid) comments, but
reformats the file.

## Tests
<!-- How is this tested? -->
- `make test`
- `go run main.go auth login --account-id XXX --host
https://accounts.cloud.databricks.com/`
- `go run main.go auth token --account-id XXX --host
https://accounts.cloud.databricks.com/`
- `go run main.go auth login --host https://XXX.cloud.databricks.com/`
2023-06-02 13:49:39 +02:00
shreyas-goenka 91097856b5
Add check for path is a directory in filer.ReadDir (#426)
## Tests
Integration tests
2023-06-02 12:28:35 +02:00
Pieter Noordhuis 2b56af6016
Add Stat function to filer.Filer interface (#421)
## Changes

TSIA

## Tests

New integration test passes.
2023-06-01 20:23:22 +02:00
Pieter Noordhuis 7a4ca786d8
Use cmdio in version command for `--output` flag (#419)
## Changes

Use cmdio in the version command such that it accepts the `--output` flag.

This removes the existing `--detail` flag which previously made the
command print JSON output.

## Tests

New integration test passes.
2023-06-01 12:03:22 +02:00
Serge Smertin 24ebfdf31e
Add readable console logger (#370)
## Changes

Add a readable colored console logger that is active only for TTYs:

<img width="764" alt="image"
src="https://user-images.githubusercontent.com/259697/235221427-ca482b32-9f88-4adb-ada3-8c4f35f50f06.png">

## Tests

Run `go run main.go clusters list --log-level debug --profile demo`
2023-06-01 11:37:33 +02:00
dependabot[bot] d44d8ff2dc
Bump github.com/mattn/go-isatty from 0.0.18 to 0.0.19 (#412) 2023-06-01 09:45:54 +02:00
Pieter Noordhuis 9ae86e3ae3
Fix locker integration test (#417)
## Changes

The failure was caused by swapping out the error types returned by the filer in #139.

## Tests

Integration tests pass again.
2023-06-01 09:38:03 +02:00
Pieter Noordhuis 349e2aff40
Allow equivalence checking of filer errors to fs errors (#416)
## Changes

The pattern `errors.Is(err, fs.ErrNotExist)` is common to check for an
error type.

Errors can implement `Is(error) bool` with a custom equivalence checker.

## Tests

New asserts all pass in the integration test.
2023-05-31 20:47:00 +02:00
Pieter Noordhuis 42cd8daee0
Make filer.Filer return fs.DirEntry from ReadDir (#415)
## Changes

This allows for compatibility with the stdlib functions in io/fs.

## Tests

Integration tests pass.
2023-05-31 14:22:26 +02:00
Pieter Noordhuis 27df4e765c
Implement DBFS filer (#139)
Adds a DBFS implementation of the `filer.Filer` interface.

The integration tests are reused between the workspace filesystem and
DBFS implementations to ensure identical behavior.
2023-05-31 13:24:20 +02:00
Pieter Noordhuis 92cb52041d
Add Mkdir and ReadDir functions to filer.Filer interface (#414)
## Changes

This cherry-picks the filer changes from #408.

## Tests

Manually ran integration tests.
2023-05-31 11:11:17 +02:00
Andrew Nester 05eaf7ff50
Added secrets input prompt for secrets put-secret command (#413)
## Changes
Added secrets input prompt for secrets put-secrets command

## Tests

<img width="623" alt="Screenshot 2023-05-30 at 12 06 24"
src="https://github.com/databricks/cli/assets/2969996/9338e6ba-c504-48cc-ac97-cac97dde7a3a">
2023-05-31 10:18:29 +02:00
Andrew Nester aed6450baf
Do not prompt for List methods (#411)
## Changes
Do not prompt for List methods

## Tests
Running
```
cli workspace list
```

Before

```
cli workspace list
Error: Path () doesn't start with '/'
```

After

```
cli workspace list
Error: accepts 1 arg(s), received 0

```
2023-05-26 16:02:53 +02:00
Andrew Nester 3c4d6f637f
Changed service template to correctly handle required positional arguments (#405) 2023-05-26 14:46:08 +02:00