Commit Graph

472 Commits

Author SHA1 Message Date
Pieter Noordhuis 854444077f
Reduce parallellism in locker integration test (#407)
## Changes

This test is hitting request throttling. A lower number of parallel
routines acquiring it should be sufficient.

## Tests

n/a
2023-05-25 15:22:51 +02:00
Andrew Nester 6141476ca2
Added support for bundle.Seq, simplified Mutator.Apply interface (#403)
## Changes
Added support for `bundle.Seq`, simplified `Mutator.Apply` interface by
removing list of mutators from return values/

## Tests
1. Ran `cli bundle deploy` and interrupted it with Cmd + C mid execution
so lock is not released
2. Ran `cli bundle deploy` top make sure that CLI is not trying to
release lock when it fail to acquire it
```
andrew.nester@HFW9Y94129 multiples-tasks % cli bundle deploy
Starting upload of bundle files
Uploaded bundle files at /Users/andrew.nester@databricks.com/.bundle/simple-task/development/files!

^C
andrew.nester@HFW9Y94129 multiples-tasks % cli bundle deploy
Error: deploy lock acquired by andrew.nester@databricks.com at 2023-05-24 12:10:23.050343 +0200 CEST. Use --force to override
```
2023-05-24 14:45:19 +02:00
Andrew Nester 273271bc59
Regenerated internal schema structs based on Terraform provider schemas (#401)
## Changes
Regenerated internal schema structs based on Terraform provider schemas

Allows to use `serverless` flag in bundle config.

## Tests
Ran `cli bundle deploy` with bundle which contains pipeline with
serverless key true
2023-05-23 19:33:24 +02:00
Shreyas Goenka c02b280691
nit 2023-05-23 19:31:02 +02:00
Shreyas Goenka 2c02fdf2fb
fixed validators for number 2023-05-23 19:25:51 +02:00
Shreyas Goenka 4fa8c9c06d
added test for gen file 2023-05-23 19:15:20 +02:00
Shreyas Goenka 95d4da6dc0
fix and added execute template tests 2023-05-23 19:02:09 +02:00
Shreyas Goenka 3626588511
remove version in testdata 2023-05-23 18:52:27 +02:00
Shreyas Goenka 2cafe2e766
fix testS 2023-05-23 18:51:08 +02:00
Shreyas Goenka 8b8320554c
remove default for config file 2023-05-23 18:49:15 +02:00
Shreyas Goenka 0b62f6d258
removed versoin from schema, this will be a separate config 2023-05-23 18:44:09 +02:00
Shreyas Goenka 4ce4b3aea1
using stdlib walk and added passing file bits 2023-05-23 15:47:48 +02:00
Pieter Noordhuis 842a559424
Release v0.100.1 (#399)
## Changes

CLI:
* Sync: Gracefully handle broken notebook files
([#398](https://github.com/databricks/cli/pull/398)).
* Add version flag to print version and exit
([#394](https://github.com/databricks/cli/pull/394)).
* Pass temporary directory environment variables to subprocesses
([#395](https://github.com/databricks/cli/pull/395)).
* Rename environment variables `BRICKS_` -> `DATABRICKS_`
([#393](https://github.com/databricks/cli/pull/393)).
* Update to Go SDK v0.9.0
([#396](https://github.com/databricks/cli/pull/396)).
2023-05-23 14:21:17 +02:00
shreyas-goenka c53ad860e6
Create tmp files in the cache dir in terraform command runs (#395)
## Changes
Passes through tmp dir related env vars to the terraform process. Incase
any of them are not set, we assign temp dir inside bundle cache dir as
the location terraform should use.

## Tests
Manually checked that these env vars do override location where
os.CreateTemp files are created
2023-05-23 13:51:15 +02:00
dependabot[bot] ebb39cc8d4
Bump github.com/hashicorp/hc-install from 0.5.1 to 0.5.2 (#387) 2023-05-23 12:12:55 +02:00
Fabian Jakobs aa85638070
Sync: Gracefully handle broken notebook files (#398)
## Changes
Ignore broken notebook files during sync.

Fixes https://github.com/databricks/databricks-vscode/issues/712
2023-05-23 12:10:15 +02:00
Fabian Jakobs cc8e9b7074
Update setup.py (#397)
Fixes https://github.com/databricks/cli/security/dependabot/7
2023-05-23 11:39:57 +02:00
Pieter Noordhuis 46df551816
Update to Go SDK v0.9.0 (#396)
## Changes

See https://github.com/databricks/databricks-sdk-go/releases/tag/v0.9.0.

## Tests

Ran integration tests manually.
2023-05-22 21:27:22 +02:00
Pieter Noordhuis d86a1f0847
Add version flag to print version and exit (#394)
## Changes

With this PR, all of the command below print version and exit:
```
$ databricks -v
Databricks CLI v0.100.1-dev+4d3fa76
$ databricks --version
Databricks CLI v0.100.1-dev+4d3fa76
$ databricks version  
Databricks CLI v0.100.1-dev+4d3fa76
```

## Tests

Added integration test for each flag or command.
2023-05-22 20:55:42 +02:00
Fabian Jakobs 055e528173
Rename: bricks -> databricks (#393)
## Changes
related to https://github.com/databricks/databricks-vscode/pull/721

## Rename env vars

`BRICKS_CLI_PATH` -> `DATABRICKS_CLI_PATH`
`BRICKS_OUTPUT_FORMAT` -> `DATABRICKS_OUTPUT_FORMAT`
`BRICKS_LOG_FILE` -> `DATABRICKS_LOG_FILE`
`BRICKS_LOG_LEVEL` -> `DATABRICKS_LOG_LEVEL`
`BRICKS_LOG_FORMAT` -> `DATABRICKS_LOG_FORMAT`
`BRICKS_PROGRESS_FORMAT` -> `DATABRICKS_CLI_PROGRESS_FORMAT`
`BRICKS_UPSTREAM` -> `DATABRICKS_CLI_UPSTREAM`
`BRICKS_UPSTREAM_VERSION` -> `DATABRICKS_CLI_UPSTREAM_VERSION`
2023-05-22 16:40:50 +02:00
Shreyas Goenka 74627160cb
- 2023-05-22 12:31:18 +02:00
Shreyas Goenka 601e16e9e8
add version of schema 2023-05-22 11:06:10 +02:00
Shreyas Goenka 89d02cda6b
nit 2023-05-22 10:48:32 +02:00
Shreyas Goenka 8a9e59df3e
add tests for individual validators 2023-05-22 10:43:43 +02:00
Shreyas Goenka 2125f256e5
lint 2023-05-22 10:30:27 +02:00
Shreyas Goenka 2511c18352
add test for dir gen skip 2023-05-22 10:16:40 +02:00
Shreyas Goenka f6fc63a09e
lazy generate dirs 2023-05-22 09:46:26 +02:00
Shreyas Goenka 479e2b4e32
Add config-file flag 2023-05-22 05:58:05 +02:00
Shreyas Goenka 350053487a
added flag for config file location 2023-05-21 22:48:38 +02:00
Shreyas Goenka a21fbdd383
refactor logic out of the cmd 2023-05-21 22:37:38 +02:00
Shreyas Goenka f5384fc5ed
added a map for validators, and moved init under bundle 2023-05-19 15:39:55 +02:00
Shreyas Goenka 9a8205b3df
address comments plus close file 2023-05-17 15:33:04 +02:00
Shreyas Goenka ecc07e893f
use errors is 2023-05-17 15:27:03 +02:00
Shreyas Goenka 9a70861e4b
address some comments 2023-05-17 15:00:27 +02:00
Shreyas Goenka 5771b62afd
add close for files 2023-05-17 14:49:03 +02:00
Shreyas Goenka fb2da25f28
Add tests for validate config 2023-05-17 14:42:24 +02:00
Shreyas Goenka 1d5e09bd39
correct validateType to use reflection and added test for it 2023-05-17 14:26:35 +02:00
Shreyas Goenka a086574a69
merge complete 2023-05-17 14:07:57 +02:00
Shreyas Goenka 4b37b94630
merge after rename 2023-05-17 14:06:31 +02:00
Shreyas Goenka 742edb5215
added unknown type test 2023-05-17 14:05:06 +02:00
Shreyas Goenka 05b3b6ca2e
added tests for castfloat 2023-05-17 14:02:41 +02:00
Shreyas Goenka 970d05157c
nit 2023-05-17 13:43:27 +02:00
Shreyas Goenka 603a68bade
Added e2e tests 2023-05-17 13:41:15 +02:00
Shreyas Goenka 4b7a606d72
move walker to a separate library 2023-05-17 11:53:13 +02:00
Shreyas Goenka 9ef3547a3a
First draft 2023-05-17 02:43:41 +02:00
Pieter Noordhuis cb78644e3c
Release v0.100.0 (#391)
## Changes

This release bumps the minor version to 100 to disambiguate between
Databricks CLI "v1" (the Python version)
and this version, Databricks CLI "v2". This release is a major rewrite
of the CLI, and is not backwards compatible.

CLI:
* Rename bricks -> databricks
([#389](https://github.com/databricks/cli/pull/389)).

Bundles:
* Added ability for deferred mutator execution
([#380](https://github.com/databricks/cli/pull/380)).
* Do not truncate local state file when pulling remote changes
([#382](https://github.com/databricks/cli/pull/382)).
2023-05-16 19:44:47 +02:00
Pieter Noordhuis 8979ed1394
Fix tests for new repository name (#390) 2023-05-16 19:02:07 +02:00
Pieter Noordhuis 98ebb78c9b
Rename bricks -> databricks (#389)
## Changes

Rename all instances of "bricks" to "databricks".

## Tests

* Confirmed the goreleaser build works, uses the correct new binary
name, and produces the right archives.
* Help output is confirmed to be correct.
* Output of `git grep -w bricks` is minimal with a couple changes
remaining for after the repository rename.
2023-05-16 18:35:39 +02:00
Andrew Nester 180dfc9a40
Added ability for deferred mutator execution (#380)
## Changes
Added `DeferredMutator` and `bundle.Defer` function which allows to
always execute some mutators either in the end of execution chain or
after error occurs in the middle of execution chain.

Usage as follows:

```
deferredMutator := bundle.Defer([]bundle.Mutator{
    lock.Acquire()
    transform.DoSomething(),
    //...
}, []bundle.Mutator{
    lock.Release(),
})
```
In such case `lock.Release()` will always be executed: either when all
operations above succeed or when any of them fails

## Tests
Before the change

```
andrew.nester@HFW9Y94129 multiples-tasks % bricks bundle deploy
Starting upload of bundle files
Uploaded bundle files at /Users/andrew.nester@databricks.com/.bundle/simple-task/development/files!

Error: terraform not initialized
andrew.nester@HFW9Y94129 multiples-tasks % bricks bundle deploy
Error: deploy lock acquired by andrew.nester@databricks.com at 2023-05-10 16:41:22.902659 +0200 CEST. Use --force to override

```

After the change
```
andrew.nester@HFW9Y94129 multiples-tasks % bricks bundle deploy 
Starting upload of bundle files
Uploaded bundle files at /Users/andrew.nester@databricks.com/.bundle/simple-task/development/files!

Error: terraform not initialized
andrew.nester@HFW9Y94129 multiples-tasks % bricks bundle deploy
Starting upload of bundle files
Uploaded bundle files at /Users/andrew.nester@databricks.com/.bundle/simple-task/development/files!

Error: terraform not initialized
```
2023-05-16 18:01:50 +02:00
Andrew Nester 33fb0b3c40
Do not truncate local state file when pulling remote changes (#382)
## Changes
When local state file exists it won't be override by remote state file

## Tests
Running `bricks bundle deploy` after state push failed does not override
local state file

Use cases verified:
1. Local state file is newer than remote
2. Local state file is older than remote
3. Local state file does not exist
4. Local state file corrupted
2023-05-16 17:02:33 +02:00