Commit Graph

360 Commits

Author SHA1 Message Date
Shreyas Goenka 2c02fdf2fb
fixed validators for number 2023-05-23 19:25:51 +02:00
Shreyas Goenka 4fa8c9c06d
added test for gen file 2023-05-23 19:15:20 +02:00
Shreyas Goenka 95d4da6dc0
fix and added execute template tests 2023-05-23 19:02:09 +02:00
Shreyas Goenka 3626588511
remove version in testdata 2023-05-23 18:52:27 +02:00
Shreyas Goenka 2cafe2e766
fix testS 2023-05-23 18:51:08 +02:00
Shreyas Goenka 8b8320554c
remove default for config file 2023-05-23 18:49:15 +02:00
Shreyas Goenka 0b62f6d258
removed versoin from schema, this will be a separate config 2023-05-23 18:44:09 +02:00
Shreyas Goenka 4ce4b3aea1
using stdlib walk and added passing file bits 2023-05-23 15:47:48 +02:00
Shreyas Goenka 74627160cb
- 2023-05-22 12:31:18 +02:00
Shreyas Goenka 601e16e9e8
add version of schema 2023-05-22 11:06:10 +02:00
Shreyas Goenka 89d02cda6b
nit 2023-05-22 10:48:32 +02:00
Shreyas Goenka 8a9e59df3e
add tests for individual validators 2023-05-22 10:43:43 +02:00
Shreyas Goenka 2125f256e5
lint 2023-05-22 10:30:27 +02:00
Shreyas Goenka 2511c18352
add test for dir gen skip 2023-05-22 10:16:40 +02:00
Shreyas Goenka f6fc63a09e
lazy generate dirs 2023-05-22 09:46:26 +02:00
Shreyas Goenka 479e2b4e32
Add config-file flag 2023-05-22 05:58:05 +02:00
Shreyas Goenka 350053487a
added flag for config file location 2023-05-21 22:48:38 +02:00
Shreyas Goenka a21fbdd383
refactor logic out of the cmd 2023-05-21 22:37:38 +02:00
Shreyas Goenka f5384fc5ed
added a map for validators, and moved init under bundle 2023-05-19 15:39:55 +02:00
Shreyas Goenka 9a8205b3df
address comments plus close file 2023-05-17 15:33:04 +02:00
Shreyas Goenka ecc07e893f
use errors is 2023-05-17 15:27:03 +02:00
Shreyas Goenka 9a70861e4b
address some comments 2023-05-17 15:00:27 +02:00
Shreyas Goenka 5771b62afd
add close for files 2023-05-17 14:49:03 +02:00
Shreyas Goenka fb2da25f28
Add tests for validate config 2023-05-17 14:42:24 +02:00
Shreyas Goenka 1d5e09bd39
correct validateType to use reflection and added test for it 2023-05-17 14:26:35 +02:00
Shreyas Goenka a086574a69
merge complete 2023-05-17 14:07:57 +02:00
Shreyas Goenka 4b37b94630
merge after rename 2023-05-17 14:06:31 +02:00
Shreyas Goenka 742edb5215
added unknown type test 2023-05-17 14:05:06 +02:00
Shreyas Goenka 05b3b6ca2e
added tests for castfloat 2023-05-17 14:02:41 +02:00
Shreyas Goenka 970d05157c
nit 2023-05-17 13:43:27 +02:00
Shreyas Goenka 603a68bade
Added e2e tests 2023-05-17 13:41:15 +02:00
Shreyas Goenka 4b7a606d72
move walker to a separate library 2023-05-17 11:53:13 +02:00
Shreyas Goenka 9ef3547a3a
First draft 2023-05-17 02:43:41 +02:00
Pieter Noordhuis cb78644e3c
Release v0.100.0 (#391)
## Changes

This release bumps the minor version to 100 to disambiguate between
Databricks CLI "v1" (the Python version)
and this version, Databricks CLI "v2". This release is a major rewrite
of the CLI, and is not backwards compatible.

CLI:
* Rename bricks -> databricks
([#389](https://github.com/databricks/cli/pull/389)).

Bundles:
* Added ability for deferred mutator execution
([#380](https://github.com/databricks/cli/pull/380)).
* Do not truncate local state file when pulling remote changes
([#382](https://github.com/databricks/cli/pull/382)).
2023-05-16 19:44:47 +02:00
Pieter Noordhuis 8979ed1394
Fix tests for new repository name (#390) 2023-05-16 19:02:07 +02:00
Pieter Noordhuis 98ebb78c9b
Rename bricks -> databricks (#389)
## Changes

Rename all instances of "bricks" to "databricks".

## Tests

* Confirmed the goreleaser build works, uses the correct new binary
name, and produces the right archives.
* Help output is confirmed to be correct.
* Output of `git grep -w bricks` is minimal with a couple changes
remaining for after the repository rename.
2023-05-16 18:35:39 +02:00
Andrew Nester 180dfc9a40
Added ability for deferred mutator execution (#380)
## Changes
Added `DeferredMutator` and `bundle.Defer` function which allows to
always execute some mutators either in the end of execution chain or
after error occurs in the middle of execution chain.

Usage as follows:

```
deferredMutator := bundle.Defer([]bundle.Mutator{
    lock.Acquire()
    transform.DoSomething(),
    //...
}, []bundle.Mutator{
    lock.Release(),
})
```
In such case `lock.Release()` will always be executed: either when all
operations above succeed or when any of them fails

## Tests
Before the change

```
andrew.nester@HFW9Y94129 multiples-tasks % bricks bundle deploy
Starting upload of bundle files
Uploaded bundle files at /Users/andrew.nester@databricks.com/.bundle/simple-task/development/files!

Error: terraform not initialized
andrew.nester@HFW9Y94129 multiples-tasks % bricks bundle deploy
Error: deploy lock acquired by andrew.nester@databricks.com at 2023-05-10 16:41:22.902659 +0200 CEST. Use --force to override

```

After the change
```
andrew.nester@HFW9Y94129 multiples-tasks % bricks bundle deploy 
Starting upload of bundle files
Uploaded bundle files at /Users/andrew.nester@databricks.com/.bundle/simple-task/development/files!

Error: terraform not initialized
andrew.nester@HFW9Y94129 multiples-tasks % bricks bundle deploy
Starting upload of bundle files
Uploaded bundle files at /Users/andrew.nester@databricks.com/.bundle/simple-task/development/files!

Error: terraform not initialized
```
2023-05-16 18:01:50 +02:00
Andrew Nester 33fb0b3c40
Do not truncate local state file when pulling remote changes (#382)
## Changes
When local state file exists it won't be override by remote state file

## Tests
Running `bricks bundle deploy` after state push failed does not override
local state file

Use cases verified:
1. Local state file is newer than remote
2. Local state file is older than remote
3. Local state file does not exist
4. Local state file corrupted
2023-05-16 17:02:33 +02:00
shreyas-goenka 2786ec85aa
Release v0.0.32 (#388)
## Changes
## 0.0.32
* Add support for variables in bundle config. Introduces 4 ways of
setting variable values, which in decreasing order of priority are:
([#383](https://github.com/databricks/bricks/pull/383))([#359](https://github.com/databricks/bricks/pull/359)).
	1. Command line flag. For example: `--var="foo=bar"`
	2. Environment variable. eg: BUNDLE_VAR_foo=bar
	3. Default value as defined in the applicable environments block
	4. Default value defined in variable definition
* Make the git details bundle config block optional
([#372](https://github.com/databricks/bricks/pull/372)).
* Fix api post integration tests
([#371](https://github.com/databricks/bricks/pull/371)).
* Fix table of content by removing not required top-level item
([#366](https://github.com/databricks/bricks/pull/366)).
* Fix printing the tasks in job output in DAG execution order
([#377](https://github.com/databricks/bricks/pull/377)).
* Improved error message when 'bricks bundle run' is executed before
'bricks bundle deploy'
([#378](https://github.com/databricks/bricks/pull/378)).
2023-05-15 16:37:47 +02:00
shreyas-goenka dd04875ee9
Add config environment support for variable overriding (#383)
## Changes
Allows to override default value for a variable definition from the
environment block in a bundle config. See bundle.yml for example usage

## Tests
Unit tests

---------

Co-authored-by: Pieter Noordhuis <pieter.noordhuis@databricks.com>
2023-05-15 14:07:18 +02:00
shreyas-goenka c5e940f664
Add support for variables in bundle config (#359)
## Changes
This PR now allows you to define variables in the bundle config and set
them in three ways
1. command line args
2. process environment variable
3. in the bundle config itself

## Tests
manually, unit, and black box tests

---------

Co-authored-by: Miles Yucht <miles@databricks.com>
2023-05-15 11:34:05 +02:00
Shreyas Goenka 1a431dd26c
wip 2023-05-15 11:25:01 +02:00
Andrew Nester c98b8dd583
Revert "Bump github.com/imdario/mergo from 0.3.13 to 0.3.15" (#381)
Reverts databricks/bricks#281
2023-05-10 18:02:10 +02:00
dependabot[bot] a52c0ee973
Bump github.com/imdario/mergo from 0.3.13 to 0.3.15 (#281)
Bumps [github.com/imdario/mergo](https://github.com/imdario/mergo) from
0.3.13 to 0.3.15.
<details>
<summary>Commits</summary>
<ul>
<li><a
href="fb3c082091"><code>fb3c082</code></a>
Update stackaid dependencies for imdario/mergo</li>
<li><a
href="c9e40b59db"><code>c9e40b5</code></a>
Merge pull request <a
href="https://redirect.github.com/imdario/mergo/issues/227">#227</a>
from eh-steve/should-not-dereference</li>
<li><a
href="acd16ec124"><code>acd16ec</code></a>
Merge branch 'master' into should-not-dereference</li>
<li><a
href="62d1cf2897"><code>62d1cf2</code></a>
Update stackaid dependencies for imdario/mergo</li>
<li><a
href="dcef160919"><code>dcef160</code></a>
Merge pull request <a
href="https://redirect.github.com/imdario/mergo/issues/226">#226</a>
from ronaudinho/fix/202</li>
<li><a
href="404749e66d"><code>404749e</code></a>
Update stackaid dependencies for imdario/mergo</li>
<li><a
href="bd316d34ac"><code>bd316d3</code></a>
Merge pull request <a
href="https://redirect.github.com/imdario/mergo/issues/221">#221</a>
from zhiyu0729/issue220</li>
<li><a
href="5b9bbdb8b6"><code>5b9bbdb</code></a>
Update stackaid dependencies for imdario/mergo</li>
<li><a
href="90db0ad2f4"><code>90db0ad</code></a>
Merge pull request <a
href="https://redirect.github.com/imdario/mergo/issues/170">#170</a>
from mdeltour/fix/naming-typo</li>
<li><a
href="d19834bb68"><code>d19834b</code></a>
Update stackaid dependencies for imdario/mergo</li>
<li>Additional commits viewable in <a
href="https://github.com/imdario/mergo/compare/v0.3.13...v0.3.15">compare
view</a></li>
</ul>
</details>
<br />


[![Dependabot compatibility
score](https://dependabot-badges.githubapp.com/badges/compatibility_score?dependency-name=github.com/imdario/mergo&package-manager=go_modules&previous-version=0.3.13&new-version=0.3.15)](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores)

You can trigger a rebase of this PR by commenting `@dependabot rebase`.

[//]: # (dependabot-automerge-start)
[//]: # (dependabot-automerge-end)

---

<details>
<summary>Dependabot commands and options</summary>
<br />

You can trigger Dependabot actions by commenting on this PR:
- `@dependabot rebase` will rebase this PR
- `@dependabot recreate` will recreate this PR, overwriting any edits
that have been made to it
- `@dependabot merge` will merge this PR after your CI passes on it
- `@dependabot squash and merge` will squash and merge this PR after
your CI passes on it
- `@dependabot cancel merge` will cancel a previously requested merge
and block automerging
- `@dependabot reopen` will reopen this PR if it is closed
- `@dependabot close` will close this PR and stop Dependabot recreating
it. You can achieve the same result by closing it manually
- `@dependabot ignore this major version` will close this PR and stop
Dependabot creating any more for this major version (unless you reopen
the PR or upgrade to it yourself)
- `@dependabot ignore this minor version` will close this PR and stop
Dependabot creating any more for this minor version (unless you reopen
the PR or upgrade to it yourself)
- `@dependabot ignore this dependency` will close this PR and stop
Dependabot creating any more for this dependency (unless you reopen the
PR or upgrade to it yourself)


</details>> **Note**
> Automatic rebases have been disabled on this pull request as it has
been open for over 30 days.

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2023-05-10 17:56:27 +02:00
dependabot[bot] 6f3096d5c8
Bump github.com/mattn/go-isatty from 0.0.17 to 0.0.18 (#373)
Bumps [github.com/mattn/go-isatty](https://github.com/mattn/go-isatty)
from 0.0.17 to 0.0.18.
<details>
<summary>Commits</summary>
<ul>
<li><a
href="72a590e157"><code>72a590e</code></a>
Merge pull request <a
href="https://redirect.github.com/mattn/go-isatty/issues/82">#82</a>
from dolmen-go/upgrade-x-sys-tagged</li>
<li><a
href="c84e2ccda8"><code>c84e2cc</code></a>
Merge pull request <a
href="https://redirect.github.com/mattn/go-isatty/issues/83">#83</a>
from dolmen-go/ci-add-go1.20</li>
<li><a
href="77d7704101"><code>77d7704</code></a>
CI: add Go 1.20</li>
<li><a
href="1f921902b3"><code>1f92190</code></a>
CI: drop Go 1.15, 1.16</li>
<li><a
href="ddec212a9a"><code>ddec212</code></a>
deps: use a tagged version of golang.org/x/sys</li>
<li>See full diff in <a
href="https://github.com/mattn/go-isatty/compare/v0.0.17...v0.0.18">compare
view</a></li>
</ul>
</details>
<br />


[![Dependabot compatibility
score](https://dependabot-badges.githubapp.com/badges/compatibility_score?dependency-name=github.com/mattn/go-isatty&package-manager=go_modules&previous-version=0.0.17&new-version=0.0.18)](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores)

Dependabot will resolve any conflicts with this PR as long as you don't
alter it yourself. You can also trigger a rebase manually by commenting
`@dependabot rebase`.

[//]: # (dependabot-automerge-start)
[//]: # (dependabot-automerge-end)

---

<details>
<summary>Dependabot commands and options</summary>
<br />

You can trigger Dependabot actions by commenting on this PR:
- `@dependabot rebase` will rebase this PR
- `@dependabot recreate` will recreate this PR, overwriting any edits
that have been made to it
- `@dependabot merge` will merge this PR after your CI passes on it
- `@dependabot squash and merge` will squash and merge this PR after
your CI passes on it
- `@dependabot cancel merge` will cancel a previously requested merge
and block automerging
- `@dependabot reopen` will reopen this PR if it is closed
- `@dependabot close` will close this PR and stop Dependabot recreating
it. You can achieve the same result by closing it manually
- `@dependabot ignore this major version` will close this PR and stop
Dependabot creating any more for this major version (unless you reopen
the PR or upgrade to it yourself)
- `@dependabot ignore this minor version` will close this PR and stop
Dependabot creating any more for this minor version (unless you reopen
the PR or upgrade to it yourself)
- `@dependabot ignore this dependency` will close this PR and stop
Dependabot creating any more for this dependency (unless you reopen the
PR or upgrade to it yourself)


</details>

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2023-05-10 17:29:37 +02:00
dependabot[bot] cd2ea3f717
Bump golang.org/x/sync from 0.1.0 to 0.2.0 (#376)
Bumps [golang.org/x/sync](https://github.com/golang/sync) from 0.1.0 to
0.2.0.
<details>
<summary>Commits</summary>
<ul>
<li><a
href="1ea32573dd"><code>1ea3257</code></a>
singleflight: make the check for exec support in TestPanicDoChan
platform-agn...</li>
<li>See full diff in <a
href="https://github.com/golang/sync/compare/v0.1.0...v0.2.0">compare
view</a></li>
</ul>
</details>
<br />


[![Dependabot compatibility
score](https://dependabot-badges.githubapp.com/badges/compatibility_score?dependency-name=golang.org/x/sync&package-manager=go_modules&previous-version=0.1.0&new-version=0.2.0)](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores)

Dependabot will resolve any conflicts with this PR as long as you don't
alter it yourself. You can also trigger a rebase manually by commenting
`@dependabot rebase`.

[//]: # (dependabot-automerge-start)
[//]: # (dependabot-automerge-end)

---

<details>
<summary>Dependabot commands and options</summary>
<br />

You can trigger Dependabot actions by commenting on this PR:
- `@dependabot rebase` will rebase this PR
- `@dependabot recreate` will recreate this PR, overwriting any edits
that have been made to it
- `@dependabot merge` will merge this PR after your CI passes on it
- `@dependabot squash and merge` will squash and merge this PR after
your CI passes on it
- `@dependabot cancel merge` will cancel a previously requested merge
and block automerging
- `@dependabot reopen` will reopen this PR if it is closed
- `@dependabot close` will close this PR and stop Dependabot recreating
it. You can achieve the same result by closing it manually
- `@dependabot ignore this major version` will close this PR and stop
Dependabot creating any more for this major version (unless you reopen
the PR or upgrade to it yourself)
- `@dependabot ignore this minor version` will close this PR and stop
Dependabot creating any more for this minor version (unless you reopen
the PR or upgrade to it yourself)
- `@dependabot ignore this dependency` will close this PR and stop
Dependabot creating any more for this dependency (unless you reopen the
PR or upgrade to it yourself)


</details>

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2023-05-10 17:28:21 +02:00
Andrew Nester 473d2bf503
Improved error message when 'bricks bundle run' is executed before 'bricks bundle deploy' (#378)
## Changes
Improved error message when 'bricks bundle run' is executed before
'bricks bundle deploy'

The error happens when we attempt to load terraform state when it does
not exist.

The best way to check if terraform state actually exists is to call
`terraform show -json` and that's what already happens here

https://github.com/databricks/bricks/compare/main...error-before-deploy#diff-8c50f8c04e568397bc865b7e02d1f4ec5b18379d8d32daddfeb041035d804f5fL28

Absence of `state.Values` indicates that there is no state and likely
bundle was just never deployed.

## Tests
Ran `bricks bundle run test_job` on a new non-deployed bundle.

**Output:**

`Error: terraform show: No state. Did you forget to run 'bricks bundle
deploy'?`

Running `bricks bundle deploy && bricks bundle run test_job` succeeds.

---------

Co-authored-by: Pieter Noordhuis <pieter.noordhuis@databricks.com>
2023-05-10 11:02:25 +02:00
Andrew Nester 1916bc9d68
Fixed printing the tasks in job output in DAG execution order (#377)
Fixes #259

## Changes
Sort task output in an execution order based on task end time

## Tests
Added `TestTaskJobOutputOrderToString` unit test.
2023-05-08 16:35:47 +02:00
shreyas-goenka 37af3d5c4f
Add omitempty tag to bundle git details (#372)
## Changes
Add omit empty tag to git details. Otherwise this field becomes a
required field in the config json schema

## Tests
Tested by regenerating the json schema and checking that the git field
is now optional
2023-05-01 14:34:12 +02:00
Kartik Gupta 5fc7cd0adf
Fix api post integration tests (#371) 2023-05-01 11:10:02 +02:00