Commit Graph

444 Commits

Author SHA1 Message Date
shreyas-goenka 6ff00122ad
Add fs ls command for dbfs (#429)
## Changes
1. Adds fs ls command
2. Adds ability to define multiple templates

## Tests
Manually and integration tests
2023-06-05 17:41:30 +02:00
Andrew Nester 1f130f3722
Do not use FgWhite and FgBlack for terminal output (#435)
## Changes
Using white / black color for terminal output will lead to poorly
displayed content in either light or dark terminal backgrounds. Some
other CLIs experienced same issues
(https://github.com/qri-io/qri/pull/774)

Instead, let's just use color to highlight some of the output so it's
more compatible with different background styles

## Tests
<img width="772" alt="Screenshot 2023-06-05 at 16 05 09"
src="https://github.com/databricks/cli/assets/2969996/01790239-6a33-4059-86a8-d5117ea0b75f">

---

<img width="757" alt="Screenshot 2023-06-05 at 16 05 20"
src="https://github.com/databricks/cli/assets/2969996/ea3b9fdc-3782-4f4f-a9df-19e66af0c04f">
2023-06-05 17:30:40 +02:00
Andrew Nester 3dbf7a575a
Better error message if can not load prompts (#437)
## Changes
Better error message if can not load prompts

## Tests
Setup 2 jobs with the same name and ran `cli job get`

```
andrew.nester@HFW9Y94129 multiples-tasks % ../../cli/cli jobs get
Error: failed to load names for Jobs drop-down. Please manually specify required arguments. Original error: duplicate .Settings.Name: duplicatejob
```
2023-06-05 16:32:03 +02:00
Pieter Noordhuis 28d36577a2
Speed up sync integration tests (#428)
## Changes

This change implements:
* Channels for line-by-line output from stdout/stderr
* A function to wait for a sync step to complete (using above)
* Ensure all tests are prefixed `TestAccSync`
* Use temporary paths in WSFS instead of cloning a repo

## Tests

The same integration tests now pass in ~90 seconds (was ~250s).
2023-06-02 14:02:18 +00:00
Pieter Noordhuis 1c0d67f66c
Add fs.FS adapter for the filer interface (#422)
## Changes

This enables the use of `io/fs` functions `fs.Glob` and `fs.WalkDir`
with filers.

We can't use `fs.FS` as the standard interface instead of `filer.Filer` because:
1. It was made for reading from filesystems only, not writing
2. It doesn't take a context for the core functions

Therefore a wrapper will do.

## Tests

* Added unit tests to cover the adapter through a fake filer.
* Manually ran `fs.WalkDir` against both WSFS and DBFS filers.
2023-06-02 12:49:59 +00:00
Andrew Nester f8255f356b
Added spinner when loading command prompts (#420)
## Changes
Added spinner when loading command prompts

## Tests
```
andrew.nester@HFW9Y94129 multiples-tasks % cli jobs get-run
⡿ No RUN_ID argument specified. Loading names for Jobs drop-down.```
2023-06-02 14:13:18 +02:00
Serge Smertin a6c9533c1c
Add profile on `databricks auth login` (#423)
## Changes
- added saving profile to `~/.databrickscfg` whenever we do `databricks
auth login`.
- we either match profile by account id / canonical host or introduce
the new one from deployment name.
- fail on multiple profiles with matching accounts or workspace hosts.
- overriding `~/.databrickscfg` keeps the (valid) comments, but
reformats the file.

## Tests
<!-- How is this tested? -->
- `make test`
- `go run main.go auth login --account-id XXX --host
https://accounts.cloud.databricks.com/`
- `go run main.go auth token --account-id XXX --host
https://accounts.cloud.databricks.com/`
- `go run main.go auth login --host https://XXX.cloud.databricks.com/`
2023-06-02 13:49:39 +02:00
shreyas-goenka 91097856b5
Add check for path is a directory in filer.ReadDir (#426)
## Tests
Integration tests
2023-06-02 12:28:35 +02:00
Pieter Noordhuis 2b56af6016
Add Stat function to filer.Filer interface (#421)
## Changes

TSIA

## Tests

New integration test passes.
2023-06-01 20:23:22 +02:00
Pieter Noordhuis 7a4ca786d8
Use cmdio in version command for `--output` flag (#419)
## Changes

Use cmdio in the version command such that it accepts the `--output` flag.

This removes the existing `--detail` flag which previously made the
command print JSON output.

## Tests

New integration test passes.
2023-06-01 12:03:22 +02:00
Serge Smertin 24ebfdf31e
Add readable console logger (#370)
## Changes

Add a readable colored console logger that is active only for TTYs:

<img width="764" alt="image"
src="https://user-images.githubusercontent.com/259697/235221427-ca482b32-9f88-4adb-ada3-8c4f35f50f06.png">

## Tests

Run `go run main.go clusters list --log-level debug --profile demo`
2023-06-01 11:37:33 +02:00
dependabot[bot] d44d8ff2dc
Bump github.com/mattn/go-isatty from 0.0.18 to 0.0.19 (#412) 2023-06-01 09:45:54 +02:00
Pieter Noordhuis 9ae86e3ae3
Fix locker integration test (#417)
## Changes

The failure was caused by swapping out the error types returned by the filer in #139.

## Tests

Integration tests pass again.
2023-06-01 09:38:03 +02:00
Pieter Noordhuis 349e2aff40
Allow equivalence checking of filer errors to fs errors (#416)
## Changes

The pattern `errors.Is(err, fs.ErrNotExist)` is common to check for an
error type.

Errors can implement `Is(error) bool` with a custom equivalence checker.

## Tests

New asserts all pass in the integration test.
2023-05-31 20:47:00 +02:00
Pieter Noordhuis 42cd8daee0
Make filer.Filer return fs.DirEntry from ReadDir (#415)
## Changes

This allows for compatibility with the stdlib functions in io/fs.

## Tests

Integration tests pass.
2023-05-31 14:22:26 +02:00
Pieter Noordhuis 27df4e765c
Implement DBFS filer (#139)
Adds a DBFS implementation of the `filer.Filer` interface.

The integration tests are reused between the workspace filesystem and
DBFS implementations to ensure identical behavior.
2023-05-31 13:24:20 +02:00
Pieter Noordhuis 92cb52041d
Add Mkdir and ReadDir functions to filer.Filer interface (#414)
## Changes

This cherry-picks the filer changes from #408.

## Tests

Manually ran integration tests.
2023-05-31 11:11:17 +02:00
Andrew Nester 05eaf7ff50
Added secrets input prompt for secrets put-secret command (#413)
## Changes
Added secrets input prompt for secrets put-secrets command

## Tests

<img width="623" alt="Screenshot 2023-05-30 at 12 06 24"
src="https://github.com/databricks/cli/assets/2969996/9338e6ba-c504-48cc-ac97-cac97dde7a3a">
2023-05-31 10:18:29 +02:00
Andrew Nester aed6450baf
Do not prompt for List methods (#411)
## Changes
Do not prompt for List methods

## Tests
Running
```
cli workspace list
```

Before

```
cli workspace list
Error: Path () doesn't start with '/'
```

After

```
cli workspace list
Error: accepts 1 arg(s), received 0

```
2023-05-26 16:02:53 +02:00
Andrew Nester 3c4d6f637f
Changed service template to correctly handle required positional arguments (#405) 2023-05-26 14:46:08 +02:00
Pieter Noordhuis e366faf4a6
Release v0.100.2 (#410)
## Changes

CLI:
* Reduce parallellism in locker integration test
([#407](https://github.com/databricks/bricks/pull/407)).

Bundles:
* Don't pass synthesized TMPDIR if not already set
([#409](https://github.com/databricks/bricks/pull/409)).
* Added support for bundle.Seq, simplified Mutator.Apply interface
([#403](https://github.com/databricks/bricks/pull/403)).
* Regenerated internal schema structs based on Terraform provider
schemas ([#401](https://github.com/databricks/bricks/pull/401)).
2023-05-26 13:28:38 +02:00
Pieter Noordhuis e4ab455ea1
Don't pass synthesized TMPDIR if not already set (#409)
## Changes

On Unix systems, the default of `/tmp` always works. No need to
synthesize a path for it.

The custom TMPDIR was causing issues when used from GitHub Actions
runners.

## Tests

Confirmed manually this fixes the issue on GitHub Actions runners.
2023-05-26 13:05:30 +02:00
Pieter Noordhuis 854444077f
Reduce parallellism in locker integration test (#407)
## Changes

This test is hitting request throttling. A lower number of parallel
routines acquiring it should be sufficient.

## Tests

n/a
2023-05-25 15:22:51 +02:00
Andrew Nester 6141476ca2
Added support for bundle.Seq, simplified Mutator.Apply interface (#403)
## Changes
Added support for `bundle.Seq`, simplified `Mutator.Apply` interface by
removing list of mutators from return values/

## Tests
1. Ran `cli bundle deploy` and interrupted it with Cmd + C mid execution
so lock is not released
2. Ran `cli bundle deploy` top make sure that CLI is not trying to
release lock when it fail to acquire it
```
andrew.nester@HFW9Y94129 multiples-tasks % cli bundle deploy
Starting upload of bundle files
Uploaded bundle files at /Users/andrew.nester@databricks.com/.bundle/simple-task/development/files!

^C
andrew.nester@HFW9Y94129 multiples-tasks % cli bundle deploy
Error: deploy lock acquired by andrew.nester@databricks.com at 2023-05-24 12:10:23.050343 +0200 CEST. Use --force to override
```
2023-05-24 14:45:19 +02:00
Andrew Nester 273271bc59
Regenerated internal schema structs based on Terraform provider schemas (#401)
## Changes
Regenerated internal schema structs based on Terraform provider schemas

Allows to use `serverless` flag in bundle config.

## Tests
Ran `cli bundle deploy` with bundle which contains pipeline with
serverless key true
2023-05-23 19:33:24 +02:00
Shreyas Goenka c02b280691
nit 2023-05-23 19:31:02 +02:00
Shreyas Goenka 2c02fdf2fb
fixed validators for number 2023-05-23 19:25:51 +02:00
Shreyas Goenka 4fa8c9c06d
added test for gen file 2023-05-23 19:15:20 +02:00
Shreyas Goenka 95d4da6dc0
fix and added execute template tests 2023-05-23 19:02:09 +02:00
Shreyas Goenka 3626588511
remove version in testdata 2023-05-23 18:52:27 +02:00
Shreyas Goenka 2cafe2e766
fix testS 2023-05-23 18:51:08 +02:00
Shreyas Goenka 8b8320554c
remove default for config file 2023-05-23 18:49:15 +02:00
Shreyas Goenka 0b62f6d258
removed versoin from schema, this will be a separate config 2023-05-23 18:44:09 +02:00
Shreyas Goenka 4ce4b3aea1
using stdlib walk and added passing file bits 2023-05-23 15:47:48 +02:00
Pieter Noordhuis 842a559424
Release v0.100.1 (#399)
## Changes

CLI:
* Sync: Gracefully handle broken notebook files
([#398](https://github.com/databricks/cli/pull/398)).
* Add version flag to print version and exit
([#394](https://github.com/databricks/cli/pull/394)).
* Pass temporary directory environment variables to subprocesses
([#395](https://github.com/databricks/cli/pull/395)).
* Rename environment variables `BRICKS_` -> `DATABRICKS_`
([#393](https://github.com/databricks/cli/pull/393)).
* Update to Go SDK v0.9.0
([#396](https://github.com/databricks/cli/pull/396)).
2023-05-23 14:21:17 +02:00
shreyas-goenka c53ad860e6
Create tmp files in the cache dir in terraform command runs (#395)
## Changes
Passes through tmp dir related env vars to the terraform process. Incase
any of them are not set, we assign temp dir inside bundle cache dir as
the location terraform should use.

## Tests
Manually checked that these env vars do override location where
os.CreateTemp files are created
2023-05-23 13:51:15 +02:00
dependabot[bot] ebb39cc8d4
Bump github.com/hashicorp/hc-install from 0.5.1 to 0.5.2 (#387) 2023-05-23 12:12:55 +02:00
Fabian Jakobs aa85638070
Sync: Gracefully handle broken notebook files (#398)
## Changes
Ignore broken notebook files during sync.

Fixes https://github.com/databricks/databricks-vscode/issues/712
2023-05-23 12:10:15 +02:00
Fabian Jakobs cc8e9b7074
Update setup.py (#397)
Fixes https://github.com/databricks/cli/security/dependabot/7
2023-05-23 11:39:57 +02:00
Pieter Noordhuis 46df551816
Update to Go SDK v0.9.0 (#396)
## Changes

See https://github.com/databricks/databricks-sdk-go/releases/tag/v0.9.0.

## Tests

Ran integration tests manually.
2023-05-22 21:27:22 +02:00
Pieter Noordhuis d86a1f0847
Add version flag to print version and exit (#394)
## Changes

With this PR, all of the command below print version and exit:
```
$ databricks -v
Databricks CLI v0.100.1-dev+4d3fa76
$ databricks --version
Databricks CLI v0.100.1-dev+4d3fa76
$ databricks version  
Databricks CLI v0.100.1-dev+4d3fa76
```

## Tests

Added integration test for each flag or command.
2023-05-22 20:55:42 +02:00
Fabian Jakobs 055e528173
Rename: bricks -> databricks (#393)
## Changes
related to https://github.com/databricks/databricks-vscode/pull/721

## Rename env vars

`BRICKS_CLI_PATH` -> `DATABRICKS_CLI_PATH`
`BRICKS_OUTPUT_FORMAT` -> `DATABRICKS_OUTPUT_FORMAT`
`BRICKS_LOG_FILE` -> `DATABRICKS_LOG_FILE`
`BRICKS_LOG_LEVEL` -> `DATABRICKS_LOG_LEVEL`
`BRICKS_LOG_FORMAT` -> `DATABRICKS_LOG_FORMAT`
`BRICKS_PROGRESS_FORMAT` -> `DATABRICKS_CLI_PROGRESS_FORMAT`
`BRICKS_UPSTREAM` -> `DATABRICKS_CLI_UPSTREAM`
`BRICKS_UPSTREAM_VERSION` -> `DATABRICKS_CLI_UPSTREAM_VERSION`
2023-05-22 16:40:50 +02:00
Shreyas Goenka 74627160cb
- 2023-05-22 12:31:18 +02:00
Shreyas Goenka 601e16e9e8
add version of schema 2023-05-22 11:06:10 +02:00
Shreyas Goenka 89d02cda6b
nit 2023-05-22 10:48:32 +02:00
Shreyas Goenka 8a9e59df3e
add tests for individual validators 2023-05-22 10:43:43 +02:00
Shreyas Goenka 2125f256e5
lint 2023-05-22 10:30:27 +02:00
Shreyas Goenka 2511c18352
add test for dir gen skip 2023-05-22 10:16:40 +02:00
Shreyas Goenka f6fc63a09e
lazy generate dirs 2023-05-22 09:46:26 +02:00
Shreyas Goenka 479e2b4e32
Add config-file flag 2023-05-22 05:58:05 +02:00