Commit Graph

782 Commits

Author SHA1 Message Date
Pieter Noordhuis 96643fd746
Hide commands under preview from help output (#469)
## Changes

TSIA

Depends on https://github.com/databricks/databricks-sdk-go/pull/497.

## Tests

Manually looked at the help output and no longer see the hidden commands.
2023-06-14 14:59:57 +00:00
Serge Smertin 63437a6a98
Create LICENSE (#467)
Adding Databricks License v2
2023-06-14 14:38:52 +00:00
shreyas-goenka 968489d7c4
Restore flags to original values on test completion (#470)
## Changes

This is necessary to avoid test interference.

## Tests

Manually.

---------

Co-authored-by: Pieter Noordhuis <pieter.noordhuis@databricks.com>
2023-06-14 14:53:27 +02:00
shreyas-goenka 3d03df8844
Add mode default as a valid set value for progress-format flag (#472)
Manually tested `progress-format` works fine for all three modes
2023-06-14 13:26:56 +02:00
Pieter Noordhuis a17876480a
Include [DEFAULT] section header when writing ~/.databrickscfg (#464)
## Changes

The ini library omits the default section header and in doing so breaks
compatibility with Python's config parser. It raises:

```
Error: MissingSectionHeaderError: File contains no section headers.
```

This commit makes sure the DEFAULT section header is included.

If the config file doesn't include a DEFAULT section itself, we include
a comment describing its purpose.

## Tests

New tests pass. Manually confirmed the DEFAULT section header is
included.

---------

Co-authored-by: PaulCornellDB <paul.cornell@databricks.com>
2023-06-13 16:41:56 +00:00
Pieter Noordhuis f219a0da5a
Annotate generated commands with OpenAPI package name (#466)
Co-authored-by: Serge Smertin <259697+nfx@users.noreply.github.com>
2023-06-13 17:20:42 +02:00
shreyas-goenka d38649088c
Add workspace import-dir command (#456)
## Tests
Testing using integration tests and manually
2023-06-12 21:03:46 +02:00
Pieter Noordhuis 35dd56c716
Release v0.100.3 (#461)
## Changes

CLI:
* Add directory tracking to sync
([#425](https://github.com/databricks/cli/pull/425)).
* Add fs cat command for dbfs files
([#430](https://github.com/databricks/cli/pull/430)).
* Add fs ls command for dbfs
([#429](https://github.com/databricks/cli/pull/429)).
* Add fs mkdirs command for dbfs
([#432](https://github.com/databricks/cli/pull/432)).
* Add fs rm command for dbfs
([#433](https://github.com/databricks/cli/pull/433)).
* Add installation instructions
([#458](https://github.com/databricks/cli/pull/458)).
* Add new line to cmdio JSON rendering
([#443](https://github.com/databricks/cli/pull/443)).
* Add profile on `databricks auth login`
([#423](https://github.com/databricks/cli/pull/423)).
* Add readable console logger
([#370](https://github.com/databricks/cli/pull/370)).
* Add workspace export-dir command
([#449](https://github.com/databricks/cli/pull/449)).
* Added secrets input prompt for secrets put-secret command
([#413](https://github.com/databricks/cli/pull/413)).
* Added spinner when loading command prompts
([#420](https://github.com/databricks/cli/pull/420)).
* Better error message if can not load prompts
([#437](https://github.com/databricks/cli/pull/437)).
* Changed service template to correctly handle required positional
arguments ([#405](https://github.com/databricks/cli/pull/405)).
* Do not generate prompts for certain commands
([#438](https://github.com/databricks/cli/pull/438)).
* Do not prompt for List methods
([#411](https://github.com/databricks/cli/pull/411)).
* Do not use FgWhite and FgBlack for terminal output
([#435](https://github.com/databricks/cli/pull/435)).
* Skip path translation of job task for jobs with a Git source
([#404](https://github.com/databricks/cli/pull/404)).
* Tweak profile prompt
([#454](https://github.com/databricks/cli/pull/454)).
* Update with the latest Go SDK
([#457](https://github.com/databricks/cli/pull/457)).
* Use cmdio in version command for `--output` flag
([#419](https://github.com/databricks/cli/pull/419)).

Bundles:
* Check for nil environment before accessing it
([#453](https://github.com/databricks/cli/pull/453)).

Dependencies:
* Bump github.com/hashicorp/terraform-json from 0.16.0 to 0.17.0
([#459](https://github.com/databricks/cli/pull/459)).
* Bump github.com/mattn/go-isatty from 0.0.18 to 0.0.19
([#412](https://github.com/databricks/cli/pull/412)).

Internal:
* Add Mkdir and ReadDir functions to filer.Filer interface
([#414](https://github.com/databricks/cli/pull/414)).
* Add Stat function to filer.Filer interface
([#421](https://github.com/databricks/cli/pull/421)).
* Add check for path is a directory in filer.ReadDir
([#426](https://github.com/databricks/cli/pull/426)).
* Add fs.FS adapter for the filer interface
([#422](https://github.com/databricks/cli/pull/422)).
* Add implementation of filer.Filer for local filesystem
([#460](https://github.com/databricks/cli/pull/460)).
* Allow equivalence checking of filer errors to fs errors
([#416](https://github.com/databricks/cli/pull/416)).
* Fix locker integration test
([#417](https://github.com/databricks/cli/pull/417)).
* Implement DBFS filer
([#139](https://github.com/databricks/cli/pull/139)).
* Include recursive deletion in filer interface
([#442](https://github.com/databricks/cli/pull/442)).
* Make filer.Filer return fs.DirEntry from ReadDir
([#415](https://github.com/databricks/cli/pull/415)).
* Speed up sync integration tests
([#428](https://github.com/databricks/cli/pull/428)).
2023-06-12 17:08:00 +02:00
Lennart Kats (databricks) 03964537b2
Add installation instructions (#458)
## Changes

Add installation instructions to the README.
2023-06-12 16:31:22 +02:00
dependabot[bot] 2c2b8655a6
Bump github.com/hashicorp/terraform-json from 0.16.0 to 0.17.0 (#459) 2023-06-12 14:09:41 +00:00
Pieter Noordhuis 960ce2e18e
Add implementation of filer.Filer for local filesystem (#460)
## Changes

Local file reads on Windows require the file handle to be closed after
using it. This commit includes an interface change to return an
`io.ReadCloser` from `Read` to accommodate this.

## Tests

The existing integration tests for the filer interface all pass.
2023-06-12 15:53:58 +02:00
Serge Smertin 2aa61a7c1b
Update with the latest Go SDK (#457)
## Changes
- removed deprecated methods
- regenerated with the latest OpenAPI spec
- picked up the latest go SDK version

## Tests
`make test`
2023-06-12 14:23:21 +02:00
shreyas-goenka cd99fce2a5
Add fs mkdirs command for dbfs (#432)
## Changes
TSIA

## Tests
Integration tests
2023-06-12 14:05:56 +02:00
Pieter Noordhuis 16bb224108
Add directory tracking to sync (#425)
## Changes

This change replaces usage of the `repofiles` package with the `filer`
package to consolidate WSFS code paths.

The `repofiles` package implemented the following behavior. If a file at
`foo/bar.txt` was created and removed, the directory `foo` was kept
around because we do not perform directory tracking. If subsequently, a
file at `foo` was created, it resulted in an `fs.ErrExist` because it is
impossible to overwrite a directory. It would then perform a recursive
delete of the path if this happened and retry the file write.

To make this use case work without resorting to a recursive delete on
conflict, we need to implement directory tracking as part of sync. The
approach in this commit is as follows:

1. Maintain set of directories needed for current set of files. Compare
to previous set of files. This results in mkdir of added directories and
rmdir of removed directories.
2. Creation of new directories should happen prior to writing files.
Otherwise, many file writes may race to create the same parent
directories, resulting in additional API calls. Removal of existing
directories should happen after removing files.
3. Making new directories can be deduped across common prefixes where
only the longest prefix is created recursively.
4. Removing existing directories must happen sequentially, starting with
the longest prefix.
5. Removal of directories is a best effort. It fails only if the
directory is not empty, and if this happens we know something placed a
file or directory manually, outside of sync.

## Tests

* Existing integration tests pass (modified where it used to assert
directories weren't cleaned up)
* New integration test to confirm the inability to remove a directory
doesn't fail the sync run
2023-06-12 11:44:00 +00:00
Pieter Noordhuis e4415bfbcf
Tweak profile prompt (#454)
## Changes

This includes the following changes:
* Move profile loading code to libs/databrickscfg and add tests
* Update prompt label to reflect workspace/account profiles
* Start prompt in search mode by default
* Custom error if `~/.databrickscfg` doesn't exist
* Custom error if `~/.databrickscfg` doesn't contain profiles
* Use stderr for prompt so that stdout redirection works (e.g. with `jq` or `jless`)

## Tests

* New unit tests pass
* Manual tests for both workspace and account commands
* Search-by-default is really nice if you have many profiles
2023-06-09 13:56:35 +02:00
Pieter Noordhuis 894d25e434
Check for nil environment before accessing it (#453) 2023-06-08 20:55:49 +00:00
shreyas-goenka 4818541062
Add workspace export-dir command (#449)
## Changes
This PR:
1. Adds the export-dir command
2. Changes filer.Read to return an error if a user tries to read a
directory
3. Adds returning internal file structures from filer.Stat().Sys()

## Tests
Integration tests and manually
2023-06-08 18:15:12 +02:00
shreyas-goenka 53164ae880
Add new line to cmdio JSON rendering (#443)
## Changes
This PR adds a new line break to JSON rendering using cmdio. This is
useful when we call `cmdio.Render` multiple times

## Tests
Manually

Co-authored-by: Pieter Noordhuis <pieter.noordhuis@databricks.com>
2023-06-08 15:48:51 +02:00
stikkireddy 402fcdd62c
Skip path translation of job task for jobs with a Git source (#404)
## Changes

Added skipping of translating paths for notebook path in notebook tasks
and python file path in spark python tasks if the git source is not null.

Resolves: #402

## Tests

There is a unit test and also tested with a sample bundle:

```
resources:
  jobs:
    demo:
      git_source:
        git_branch: master
        git_provider: github
        git_url: https://github.com/test/dummy
   ....
```

---------

Co-authored-by: Pieter Noordhuis <pieter.noordhuis@databricks.com>
2023-06-07 12:34:59 +02:00
Pieter Noordhuis be10ff9a75
Include recursive deletion in filer interface (#442)
## Changes

This captures the recursive deletion of a directory tree in the filer interface.

Prompted by #433.

## Tests

Integration tests pass (ran the filer ones on AWS and Azure).
2023-06-06 06:27:47 +00:00
shreyas-goenka d6d35e314f
Add fs rm command for dbfs (#433)
## Changes
Please look at the title

## Tests
Integration tests
2023-06-05 23:21:47 +00:00
shreyas-goenka ae10419eb8
Add fs cat command for dbfs files (#430)
## Changes
TSIA

## Tests
Manually and integration tests
2023-06-06 01:16:23 +02:00
Andrew Nester df3f5863c7
Do not generate prompts for certain commands (#438)
## Changes
Some of the commands do not support prompts, for example `workspace
get-status` but we were wrongly suggesting customers some option.

Quick fix for this is not to provide prompts for these known commands.

Note: it uses a method from this PR in Go SDK
https://github.com/databricks/databricks-sdk-go/pull/416

## Tests
Running `workspace get-status`

Before
```
andrew.nester@HFW9Y94129 multiples-tasks % ../../cli/cli workspace get-status
Error: Path () doesn't start with '/'
```

After
```
andrew.nester@HFW9Y94129 multiples-tasks % ../../cli/cli workspace get-status
Error: accepts 1 arg(s), received 0

```
2023-06-05 19:38:45 +02:00
shreyas-goenka 6ff00122ad
Add fs ls command for dbfs (#429)
## Changes
1. Adds fs ls command
2. Adds ability to define multiple templates

## Tests
Manually and integration tests
2023-06-05 17:41:30 +02:00
Andrew Nester 1f130f3722
Do not use FgWhite and FgBlack for terminal output (#435)
## Changes
Using white / black color for terminal output will lead to poorly
displayed content in either light or dark terminal backgrounds. Some
other CLIs experienced same issues
(https://github.com/qri-io/qri/pull/774)

Instead, let's just use color to highlight some of the output so it's
more compatible with different background styles

## Tests
<img width="772" alt="Screenshot 2023-06-05 at 16 05 09"
src="https://github.com/databricks/cli/assets/2969996/01790239-6a33-4059-86a8-d5117ea0b75f">

---

<img width="757" alt="Screenshot 2023-06-05 at 16 05 20"
src="https://github.com/databricks/cli/assets/2969996/ea3b9fdc-3782-4f4f-a9df-19e66af0c04f">
2023-06-05 17:30:40 +02:00
Andrew Nester 3dbf7a575a
Better error message if can not load prompts (#437)
## Changes
Better error message if can not load prompts

## Tests
Setup 2 jobs with the same name and ran `cli job get`

```
andrew.nester@HFW9Y94129 multiples-tasks % ../../cli/cli jobs get
Error: failed to load names for Jobs drop-down. Please manually specify required arguments. Original error: duplicate .Settings.Name: duplicatejob
```
2023-06-05 16:32:03 +02:00
Pieter Noordhuis 28d36577a2
Speed up sync integration tests (#428)
## Changes

This change implements:
* Channels for line-by-line output from stdout/stderr
* A function to wait for a sync step to complete (using above)
* Ensure all tests are prefixed `TestAccSync`
* Use temporary paths in WSFS instead of cloning a repo

## Tests

The same integration tests now pass in ~90 seconds (was ~250s).
2023-06-02 14:02:18 +00:00
Pieter Noordhuis 1c0d67f66c
Add fs.FS adapter for the filer interface (#422)
## Changes

This enables the use of `io/fs` functions `fs.Glob` and `fs.WalkDir`
with filers.

We can't use `fs.FS` as the standard interface instead of `filer.Filer` because:
1. It was made for reading from filesystems only, not writing
2. It doesn't take a context for the core functions

Therefore a wrapper will do.

## Tests

* Added unit tests to cover the adapter through a fake filer.
* Manually ran `fs.WalkDir` against both WSFS and DBFS filers.
2023-06-02 12:49:59 +00:00
Andrew Nester f8255f356b
Added spinner when loading command prompts (#420)
## Changes
Added spinner when loading command prompts

## Tests
```
andrew.nester@HFW9Y94129 multiples-tasks % cli jobs get-run
⡿ No RUN_ID argument specified. Loading names for Jobs drop-down.```
2023-06-02 14:13:18 +02:00
Serge Smertin a6c9533c1c
Add profile on `databricks auth login` (#423)
## Changes
- added saving profile to `~/.databrickscfg` whenever we do `databricks
auth login`.
- we either match profile by account id / canonical host or introduce
the new one from deployment name.
- fail on multiple profiles with matching accounts or workspace hosts.
- overriding `~/.databrickscfg` keeps the (valid) comments, but
reformats the file.

## Tests
<!-- How is this tested? -->
- `make test`
- `go run main.go auth login --account-id XXX --host
https://accounts.cloud.databricks.com/`
- `go run main.go auth token --account-id XXX --host
https://accounts.cloud.databricks.com/`
- `go run main.go auth login --host https://XXX.cloud.databricks.com/`
2023-06-02 13:49:39 +02:00
shreyas-goenka 91097856b5
Add check for path is a directory in filer.ReadDir (#426)
## Tests
Integration tests
2023-06-02 12:28:35 +02:00
Pieter Noordhuis 2b56af6016
Add Stat function to filer.Filer interface (#421)
## Changes

TSIA

## Tests

New integration test passes.
2023-06-01 20:23:22 +02:00
Pieter Noordhuis 7a4ca786d8
Use cmdio in version command for `--output` flag (#419)
## Changes

Use cmdio in the version command such that it accepts the `--output` flag.

This removes the existing `--detail` flag which previously made the
command print JSON output.

## Tests

New integration test passes.
2023-06-01 12:03:22 +02:00
Serge Smertin 24ebfdf31e
Add readable console logger (#370)
## Changes

Add a readable colored console logger that is active only for TTYs:

<img width="764" alt="image"
src="https://user-images.githubusercontent.com/259697/235221427-ca482b32-9f88-4adb-ada3-8c4f35f50f06.png">

## Tests

Run `go run main.go clusters list --log-level debug --profile demo`
2023-06-01 11:37:33 +02:00
dependabot[bot] d44d8ff2dc
Bump github.com/mattn/go-isatty from 0.0.18 to 0.0.19 (#412) 2023-06-01 09:45:54 +02:00
Pieter Noordhuis 9ae86e3ae3
Fix locker integration test (#417)
## Changes

The failure was caused by swapping out the error types returned by the filer in #139.

## Tests

Integration tests pass again.
2023-06-01 09:38:03 +02:00
Pieter Noordhuis 349e2aff40
Allow equivalence checking of filer errors to fs errors (#416)
## Changes

The pattern `errors.Is(err, fs.ErrNotExist)` is common to check for an
error type.

Errors can implement `Is(error) bool` with a custom equivalence checker.

## Tests

New asserts all pass in the integration test.
2023-05-31 20:47:00 +02:00
Pieter Noordhuis 42cd8daee0
Make filer.Filer return fs.DirEntry from ReadDir (#415)
## Changes

This allows for compatibility with the stdlib functions in io/fs.

## Tests

Integration tests pass.
2023-05-31 14:22:26 +02:00
Pieter Noordhuis 27df4e765c
Implement DBFS filer (#139)
Adds a DBFS implementation of the `filer.Filer` interface.

The integration tests are reused between the workspace filesystem and
DBFS implementations to ensure identical behavior.
2023-05-31 13:24:20 +02:00
Pieter Noordhuis 92cb52041d
Add Mkdir and ReadDir functions to filer.Filer interface (#414)
## Changes

This cherry-picks the filer changes from #408.

## Tests

Manually ran integration tests.
2023-05-31 11:11:17 +02:00
Andrew Nester 05eaf7ff50
Added secrets input prompt for secrets put-secret command (#413)
## Changes
Added secrets input prompt for secrets put-secrets command

## Tests

<img width="623" alt="Screenshot 2023-05-30 at 12 06 24"
src="https://github.com/databricks/cli/assets/2969996/9338e6ba-c504-48cc-ac97-cac97dde7a3a">
2023-05-31 10:18:29 +02:00
Andrew Nester aed6450baf
Do not prompt for List methods (#411)
## Changes
Do not prompt for List methods

## Tests
Running
```
cli workspace list
```

Before

```
cli workspace list
Error: Path () doesn't start with '/'
```

After

```
cli workspace list
Error: accepts 1 arg(s), received 0

```
2023-05-26 16:02:53 +02:00
Andrew Nester 3c4d6f637f
Changed service template to correctly handle required positional arguments (#405) 2023-05-26 14:46:08 +02:00
Pieter Noordhuis e366faf4a6
Release v0.100.2 (#410)
## Changes

CLI:
* Reduce parallellism in locker integration test
([#407](https://github.com/databricks/bricks/pull/407)).

Bundles:
* Don't pass synthesized TMPDIR if not already set
([#409](https://github.com/databricks/bricks/pull/409)).
* Added support for bundle.Seq, simplified Mutator.Apply interface
([#403](https://github.com/databricks/bricks/pull/403)).
* Regenerated internal schema structs based on Terraform provider
schemas ([#401](https://github.com/databricks/bricks/pull/401)).
2023-05-26 13:28:38 +02:00
Pieter Noordhuis e4ab455ea1
Don't pass synthesized TMPDIR if not already set (#409)
## Changes

On Unix systems, the default of `/tmp` always works. No need to
synthesize a path for it.

The custom TMPDIR was causing issues when used from GitHub Actions
runners.

## Tests

Confirmed manually this fixes the issue on GitHub Actions runners.
2023-05-26 13:05:30 +02:00
Pieter Noordhuis 854444077f
Reduce parallellism in locker integration test (#407)
## Changes

This test is hitting request throttling. A lower number of parallel
routines acquiring it should be sufficient.

## Tests

n/a
2023-05-25 15:22:51 +02:00
Andrew Nester 6141476ca2
Added support for bundle.Seq, simplified Mutator.Apply interface (#403)
## Changes
Added support for `bundle.Seq`, simplified `Mutator.Apply` interface by
removing list of mutators from return values/

## Tests
1. Ran `cli bundle deploy` and interrupted it with Cmd + C mid execution
so lock is not released
2. Ran `cli bundle deploy` top make sure that CLI is not trying to
release lock when it fail to acquire it
```
andrew.nester@HFW9Y94129 multiples-tasks % cli bundle deploy
Starting upload of bundle files
Uploaded bundle files at /Users/andrew.nester@databricks.com/.bundle/simple-task/development/files!

^C
andrew.nester@HFW9Y94129 multiples-tasks % cli bundle deploy
Error: deploy lock acquired by andrew.nester@databricks.com at 2023-05-24 12:10:23.050343 +0200 CEST. Use --force to override
```
2023-05-24 14:45:19 +02:00
Andrew Nester 273271bc59
Regenerated internal schema structs based on Terraform provider schemas (#401)
## Changes
Regenerated internal schema structs based on Terraform provider schemas

Allows to use `serverless` flag in bundle config.

## Tests
Ran `cli bundle deploy` with bundle which contains pipeline with
serverless key true
2023-05-23 19:33:24 +02:00
Pieter Noordhuis 842a559424
Release v0.100.1 (#399)
## Changes

CLI:
* Sync: Gracefully handle broken notebook files
([#398](https://github.com/databricks/cli/pull/398)).
* Add version flag to print version and exit
([#394](https://github.com/databricks/cli/pull/394)).
* Pass temporary directory environment variables to subprocesses
([#395](https://github.com/databricks/cli/pull/395)).
* Rename environment variables `BRICKS_` -> `DATABRICKS_`
([#393](https://github.com/databricks/cli/pull/393)).
* Update to Go SDK v0.9.0
([#396](https://github.com/databricks/cli/pull/396)).
2023-05-23 14:21:17 +02:00
shreyas-goenka c53ad860e6
Create tmp files in the cache dir in terraform command runs (#395)
## Changes
Passes through tmp dir related env vars to the terraform process. Incase
any of them are not set, we assign temp dir inside bundle cache dir as
the location terraform should use.

## Tests
Manually checked that these env vars do override location where
os.CreateTemp files are created
2023-05-23 13:51:15 +02:00