Commit Graph

380 Commits

Author SHA1 Message Date
Shreyas Goenka da652c9957
- 2023-06-06 01:03:20 +02:00
Shreyas Goenka 5619d0fed7
- 2023-06-05 17:48:55 +02:00
Shreyas Goenka b64b7ab1a7
- 2023-06-05 17:48:21 +02:00
Shreyas Goenka 8b4f22ce95
Merge remote-tracking branch 'origin' into fs-rm 2023-06-05 17:42:09 +02:00
shreyas-goenka 6ff00122ad
Add fs ls command for dbfs (#429)
## Changes
1. Adds fs ls command
2. Adds ability to define multiple templates

## Tests
Manually and integration tests
2023-06-05 17:41:30 +02:00
Andrew Nester 1f130f3722
Do not use FgWhite and FgBlack for terminal output (#435)
## Changes
Using white / black color for terminal output will lead to poorly
displayed content in either light or dark terminal backgrounds. Some
other CLIs experienced same issues
(https://github.com/qri-io/qri/pull/774)

Instead, let's just use color to highlight some of the output so it's
more compatible with different background styles

## Tests
<img width="772" alt="Screenshot 2023-06-05 at 16 05 09"
src="https://github.com/databricks/cli/assets/2969996/01790239-6a33-4059-86a8-d5117ea0b75f">

---

<img width="757" alt="Screenshot 2023-06-05 at 16 05 20"
src="https://github.com/databricks/cli/assets/2969996/ea3b9fdc-3782-4f4f-a9df-19e66af0c04f">
2023-06-05 17:30:40 +02:00
Andrew Nester 3dbf7a575a
Better error message if can not load prompts (#437)
## Changes
Better error message if can not load prompts

## Tests
Setup 2 jobs with the same name and ran `cli job get`

```
andrew.nester@HFW9Y94129 multiples-tasks % ../../cli/cli jobs get
Error: failed to load names for Jobs drop-down. Please manually specify required arguments. Original error: duplicate .Settings.Name: duplicatejob
```
2023-06-05 16:32:03 +02:00
Shreyas Goenka 25aa5b667c
- 2023-06-05 15:42:56 +02:00
Shreyas Goenka 4b8f5746ef
Merge branch 'fs-ls' into fs-rm 2023-06-05 15:42:09 +02:00
Shreyas Goenka 9160a6e9ed
initialize dbfs client at root 2023-06-05 15:15:21 +02:00
Shreyas Goenka 08826094ce
added preallocation of size 2023-06-05 15:09:09 +02:00
Shreyas Goenka 265e517126
address comments 2 2023-06-05 15:07:08 +02:00
Shreyas Goenka b048c8675d
address comments 2023-06-05 14:01:06 +02:00
Shreyas Goenka fc7fbbec6f
- 2023-06-05 13:55:29 +02:00
Shreyas Goenka 8050dc524f
move sort outside 2023-06-05 13:53:38 +02:00
Shreyas Goenka 19a20512e9
comments 2023-06-05 13:49:33 +02:00
Shreyas Goenka 54a6602aae
Add fs rm command for dbfs 2023-06-05 03:07:19 +02:00
Shreyas Goenka 3dc47709d0
manually check for dbfs prefix 2023-06-05 02:13:28 +02:00
Shreyas Goenka 492382caa1
added resolvedbfs path func 2023-06-05 01:34:28 +02:00
Shreyas Goenka db953acb2f
Merge remote-tracking branch 'origin' into fs-ls 2023-06-05 01:21:39 +02:00
Shreyas Goenka 6b667b5e01
added test for ls on flie 2023-06-05 01:20:54 +02:00
Shreyas Goenka 2706bebb38
lint 2023-06-05 01:10:15 +02:00
Shreyas Goenka a4879c0d91
added integration tests 2023-06-05 01:06:42 +02:00
Shreyas Goenka b5f5f1103f
Add fs ls command for dbfs 2023-06-02 17:24:04 +02:00
Pieter Noordhuis 28d36577a2
Speed up sync integration tests (#428)
## Changes

This change implements:
* Channels for line-by-line output from stdout/stderr
* A function to wait for a sync step to complete (using above)
* Ensure all tests are prefixed `TestAccSync`
* Use temporary paths in WSFS instead of cloning a repo

## Tests

The same integration tests now pass in ~90 seconds (was ~250s).
2023-06-02 14:02:18 +00:00
Pieter Noordhuis 1c0d67f66c
Add fs.FS adapter for the filer interface (#422)
## Changes

This enables the use of `io/fs` functions `fs.Glob` and `fs.WalkDir`
with filers.

We can't use `fs.FS` as the standard interface instead of `filer.Filer` because:
1. It was made for reading from filesystems only, not writing
2. It doesn't take a context for the core functions

Therefore a wrapper will do.

## Tests

* Added unit tests to cover the adapter through a fake filer.
* Manually ran `fs.WalkDir` against both WSFS and DBFS filers.
2023-06-02 12:49:59 +00:00
Andrew Nester f8255f356b
Added spinner when loading command prompts (#420)
## Changes
Added spinner when loading command prompts

## Tests
```
andrew.nester@HFW9Y94129 multiples-tasks % cli jobs get-run
⡿ No RUN_ID argument specified. Loading names for Jobs drop-down.```
2023-06-02 14:13:18 +02:00
Serge Smertin a6c9533c1c
Add profile on `databricks auth login` (#423)
## Changes
- added saving profile to `~/.databrickscfg` whenever we do `databricks
auth login`.
- we either match profile by account id / canonical host or introduce
the new one from deployment name.
- fail on multiple profiles with matching accounts or workspace hosts.
- overriding `~/.databrickscfg` keeps the (valid) comments, but
reformats the file.

## Tests
<!-- How is this tested? -->
- `make test`
- `go run main.go auth login --account-id XXX --host
https://accounts.cloud.databricks.com/`
- `go run main.go auth token --account-id XXX --host
https://accounts.cloud.databricks.com/`
- `go run main.go auth login --host https://XXX.cloud.databricks.com/`
2023-06-02 13:49:39 +02:00
shreyas-goenka 91097856b5
Add check for path is a directory in filer.ReadDir (#426)
## Tests
Integration tests
2023-06-02 12:28:35 +02:00
Pieter Noordhuis 2b56af6016
Add Stat function to filer.Filer interface (#421)
## Changes

TSIA

## Tests

New integration test passes.
2023-06-01 20:23:22 +02:00
Pieter Noordhuis 7a4ca786d8
Use cmdio in version command for `--output` flag (#419)
## Changes

Use cmdio in the version command such that it accepts the `--output` flag.

This removes the existing `--detail` flag which previously made the
command print JSON output.

## Tests

New integration test passes.
2023-06-01 12:03:22 +02:00
Serge Smertin 24ebfdf31e
Add readable console logger (#370)
## Changes

Add a readable colored console logger that is active only for TTYs:

<img width="764" alt="image"
src="https://user-images.githubusercontent.com/259697/235221427-ca482b32-9f88-4adb-ada3-8c4f35f50f06.png">

## Tests

Run `go run main.go clusters list --log-level debug --profile demo`
2023-06-01 11:37:33 +02:00
dependabot[bot] d44d8ff2dc
Bump github.com/mattn/go-isatty from 0.0.18 to 0.0.19 (#412) 2023-06-01 09:45:54 +02:00
Pieter Noordhuis 9ae86e3ae3
Fix locker integration test (#417)
## Changes

The failure was caused by swapping out the error types returned by the filer in #139.

## Tests

Integration tests pass again.
2023-06-01 09:38:03 +02:00
Pieter Noordhuis 349e2aff40
Allow equivalence checking of filer errors to fs errors (#416)
## Changes

The pattern `errors.Is(err, fs.ErrNotExist)` is common to check for an
error type.

Errors can implement `Is(error) bool` with a custom equivalence checker.

## Tests

New asserts all pass in the integration test.
2023-05-31 20:47:00 +02:00
Pieter Noordhuis 42cd8daee0
Make filer.Filer return fs.DirEntry from ReadDir (#415)
## Changes

This allows for compatibility with the stdlib functions in io/fs.

## Tests

Integration tests pass.
2023-05-31 14:22:26 +02:00
Pieter Noordhuis 27df4e765c
Implement DBFS filer (#139)
Adds a DBFS implementation of the `filer.Filer` interface.

The integration tests are reused between the workspace filesystem and
DBFS implementations to ensure identical behavior.
2023-05-31 13:24:20 +02:00
Pieter Noordhuis 92cb52041d
Add Mkdir and ReadDir functions to filer.Filer interface (#414)
## Changes

This cherry-picks the filer changes from #408.

## Tests

Manually ran integration tests.
2023-05-31 11:11:17 +02:00
Andrew Nester 05eaf7ff50
Added secrets input prompt for secrets put-secret command (#413)
## Changes
Added secrets input prompt for secrets put-secrets command

## Tests

<img width="623" alt="Screenshot 2023-05-30 at 12 06 24"
src="https://github.com/databricks/cli/assets/2969996/9338e6ba-c504-48cc-ac97-cac97dde7a3a">
2023-05-31 10:18:29 +02:00
Andrew Nester aed6450baf
Do not prompt for List methods (#411)
## Changes
Do not prompt for List methods

## Tests
Running
```
cli workspace list
```

Before

```
cli workspace list
Error: Path () doesn't start with '/'
```

After

```
cli workspace list
Error: accepts 1 arg(s), received 0

```
2023-05-26 16:02:53 +02:00
Andrew Nester 3c4d6f637f
Changed service template to correctly handle required positional arguments (#405) 2023-05-26 14:46:08 +02:00
Pieter Noordhuis e366faf4a6
Release v0.100.2 (#410)
## Changes

CLI:
* Reduce parallellism in locker integration test
([#407](https://github.com/databricks/bricks/pull/407)).

Bundles:
* Don't pass synthesized TMPDIR if not already set
([#409](https://github.com/databricks/bricks/pull/409)).
* Added support for bundle.Seq, simplified Mutator.Apply interface
([#403](https://github.com/databricks/bricks/pull/403)).
* Regenerated internal schema structs based on Terraform provider
schemas ([#401](https://github.com/databricks/bricks/pull/401)).
2023-05-26 13:28:38 +02:00
Pieter Noordhuis e4ab455ea1
Don't pass synthesized TMPDIR if not already set (#409)
## Changes

On Unix systems, the default of `/tmp` always works. No need to
synthesize a path for it.

The custom TMPDIR was causing issues when used from GitHub Actions
runners.

## Tests

Confirmed manually this fixes the issue on GitHub Actions runners.
2023-05-26 13:05:30 +02:00
Pieter Noordhuis 854444077f
Reduce parallellism in locker integration test (#407)
## Changes

This test is hitting request throttling. A lower number of parallel
routines acquiring it should be sufficient.

## Tests

n/a
2023-05-25 15:22:51 +02:00
Andrew Nester 6141476ca2
Added support for bundle.Seq, simplified Mutator.Apply interface (#403)
## Changes
Added support for `bundle.Seq`, simplified `Mutator.Apply` interface by
removing list of mutators from return values/

## Tests
1. Ran `cli bundle deploy` and interrupted it with Cmd + C mid execution
so lock is not released
2. Ran `cli bundle deploy` top make sure that CLI is not trying to
release lock when it fail to acquire it
```
andrew.nester@HFW9Y94129 multiples-tasks % cli bundle deploy
Starting upload of bundle files
Uploaded bundle files at /Users/andrew.nester@databricks.com/.bundle/simple-task/development/files!

^C
andrew.nester@HFW9Y94129 multiples-tasks % cli bundle deploy
Error: deploy lock acquired by andrew.nester@databricks.com at 2023-05-24 12:10:23.050343 +0200 CEST. Use --force to override
```
2023-05-24 14:45:19 +02:00
Andrew Nester 273271bc59
Regenerated internal schema structs based on Terraform provider schemas (#401)
## Changes
Regenerated internal schema structs based on Terraform provider schemas

Allows to use `serverless` flag in bundle config.

## Tests
Ran `cli bundle deploy` with bundle which contains pipeline with
serverless key true
2023-05-23 19:33:24 +02:00
Pieter Noordhuis 842a559424
Release v0.100.1 (#399)
## Changes

CLI:
* Sync: Gracefully handle broken notebook files
([#398](https://github.com/databricks/cli/pull/398)).
* Add version flag to print version and exit
([#394](https://github.com/databricks/cli/pull/394)).
* Pass temporary directory environment variables to subprocesses
([#395](https://github.com/databricks/cli/pull/395)).
* Rename environment variables `BRICKS_` -> `DATABRICKS_`
([#393](https://github.com/databricks/cli/pull/393)).
* Update to Go SDK v0.9.0
([#396](https://github.com/databricks/cli/pull/396)).
2023-05-23 14:21:17 +02:00
shreyas-goenka c53ad860e6
Create tmp files in the cache dir in terraform command runs (#395)
## Changes
Passes through tmp dir related env vars to the terraform process. Incase
any of them are not set, we assign temp dir inside bundle cache dir as
the location terraform should use.

## Tests
Manually checked that these env vars do override location where
os.CreateTemp files are created
2023-05-23 13:51:15 +02:00
dependabot[bot] ebb39cc8d4
Bump github.com/hashicorp/hc-install from 0.5.1 to 0.5.2 (#387) 2023-05-23 12:12:55 +02:00
Fabian Jakobs aa85638070
Sync: Gracefully handle broken notebook files (#398)
## Changes
Ignore broken notebook files during sync.

Fixes https://github.com/databricks/databricks-vscode/issues/712
2023-05-23 12:10:15 +02:00