Commit Graph

352 Commits

Author SHA1 Message Date
Serge Smertin 487bf6fd5c
Use Databricks Go SDK v0.1.0 (#110)
This PR pins the version of Databricks SDK for Go to v0.1.0
2022-12-01 12:17:36 +01:00
Pieter Noordhuis 34af98a8c3
Mutators to define current user and default artifact path (#112) 2022-12-01 11:17:29 +01:00
Pieter Noordhuis 230811031f
Rename jobs -> workflows (#111) 2022-12-01 09:35:21 +01:00
Pieter Noordhuis c4d63eac70
Rudimentary interpolation support (#108)
Performs interpolation on string field.

It looks for patterns `${foo.bar}` where `foo.bar` points to a string
field in the configuration data model.

It does not support traversal (e.g. `${foo}` with `foo` equal
to`${bar}`), hence "rudimentary".
2022-12-01 09:33:42 +01:00
Pieter Noordhuis 4064a21797
Function to return bundle's cache directory (#109)
Parallel of `project.CacheDir()` introduced in
https://github.com/databricks/bricks/pull/82.
2022-11-30 14:40:41 +01:00
Pieter Noordhuis e1669b0352
Model code artifacts (#107)
This adds:
* Top level "artifacts" configuration key
* Support for notebooks (does language detection and upload)
* Merge of per-environment artifacts (or artifact overrides) into top level
2022-11-30 14:15:22 +01:00
shreyas-goenka c6b3b35e98
[DECO-396] Send delete file requests with recursive set to false (#106)
Safeguard so bugs do not delete large amount of remote files
2022-11-30 13:56:52 +01:00
shreyas-goenka 2ebfa5f369
Run unit tests on windows and macos (#103)
Unit tests are now run in all three big OS. 

Some of the changes are to make the tests green for windows while we are
skipping some of the other tests on windows/macOS to make the tests
pass. This is a temporary measure and we will incrementally migrate
these tests over so there is parity in unit testing along all three
environments!
2022-11-28 11:34:25 +01:00
Pieter Noordhuis b88b35a510
Move mutator interface to top level bundle package (#105)
While working on artifact upload and workspace interrogation I realized
this mutator interface needs to:
1. Operate at the whole bundle level so it can apply to both
configuration and internal state
2. Include a `context.Context` parameter for a) long running operations
and b) progress reporting

Previous interface:
```
Apply(*config.Root) ([]Mutator, error)
```

New interface:
```
Apply(context.Context, *Bundle) ([]Mutator, error)
```
2022-11-28 10:59:43 +01:00
Pieter Noordhuis 5c916a6fb4
Store specified environment in configuration for reference (#104) 2022-11-28 10:10:13 +01:00
shreyas-goenka 761de12a0d
Fix incorrect remote path arg to APIs in windows on bricks sync (#101)
Integration tests testing sync on ubuntu are green:
<img width="794" alt="Screenshot 2022-11-25 at 2 05 11 PM"
src="https://user-images.githubusercontent.com/88374338/203991936-f0478c35-5d97-429b-b373-f050d9a90cff.png">

@fjakobs tested this on windows manually
2022-11-25 14:13:15 +01:00
Pieter Noordhuis 8e786d76a9
Update databricks-sdk-go to latest (#102) 2022-11-24 21:41:57 +01:00
Pieter Noordhuis 07f07694a4
Function to return workspace client on bundle.Bundle (#100)
Complementary command to check the identity in the context of a bundle
environment:

For example:
```
bricks bundle debug whoami -e development
```
2022-11-23 15:20:03 +01:00
shreyas-goenka 696e337516
Update go sdk version (#97) 2022-11-22 14:52:45 +01:00
shreyas-goenka b25bd318e3
Deal with notebook remote filenames stripping .py (#90)
This PR introduces tracking of remote names and local names of files in snapshots to disambiguate between files which might have the same remote name and handle clean deleting of files whose remote name changes due (eg. python notebook getting converted to a python notebook)
2022-11-21 23:42:09 +01:00
Pieter Noordhuis ab1df558a2
Test that YAML anchors work (#96) 2022-11-21 15:40:27 +01:00
Pieter Noordhuis 3b351d3b00
Add command that writes the materialized bundle configuration to stdout (#95)
Used to inspect the bundle configuration after loading and merging all
files.

Once we add variable interpolation this command could show the result
after interpolation as well.

Each of the mutations to this configuration is observable, so we could
add a mode that writes each of the intermediate versions to disk for
even more fine grained introspection.
2022-11-21 15:39:53 +01:00
Pieter Noordhuis 195eb7f0f9
Add job and pipeline structs (#94) 2022-11-18 11:12:24 +01:00
Pieter Noordhuis e47fa61951
Skeleton for configuration loading and mutation (#92)
Load a tree of configuration files anchored at `bundle.yml` into the
`config.Root` struct.

All mutations (from setting defaults to merging files) are observable
through the `mutator.Mutator` interface.
2022-11-18 10:57:31 +01:00
Pieter Noordhuis 6a8c9f22b8
Update databricks-sdk-go to latest (#93) 2022-11-17 11:24:09 +01:00
Shreyas Goenka 0c24e6f82e
Revert "WIP initial version of the workspace file lock done"
This reverts commit 02eec1f990.
2022-11-16 23:50:17 +01:00
Shreyas Goenka 02eec1f990
WIP initial version of the workspace file lock done 2022-11-16 17:30:46 +01:00
Pieter Noordhuis af5659ae8e
Update databricks-sdk-go to latest (#89) 2022-11-09 15:01:47 +01:00
shreyas-goenka d587531cbd
Added creation of .gitignore for bricks project with cache dir path (#88)
It works, please trust me
2022-11-08 13:51:08 +01:00
Shreyas Goenka efe1f28b36
Revert "Added creation of .gitignore for bricks project with cache dir path"
This reverts commit 1505c63426.
2022-11-03 21:09:29 +01:00
Shreyas Goenka 1505c63426
Added creation of .gitignore for bricks project with cache dir path 2022-11-03 21:06:21 +01:00
shreyas-goenka 0601e114fd
[DECO-200] Ignore RESOURCE_DOES_NOT_EXIST errors on file deletion during sync (#85)
tested manually
2022-10-27 17:32:10 +02:00
shreyas-goenka 1329bc05d4
Log initial sync complete (#86)
Tested manually
<img width="930" alt="Screenshot 2022-10-27 at 2 21 52 PM"
src="https://user-images.githubusercontent.com/88374338/198282897-ad0c3d62-be92-4ec4-a0e1-4df2a7eab2b9.png">
2022-10-27 15:41:18 +02:00
shreyas-goenka 18dae73505
[DECO-79][DECO-165] Incremental sync with support for multiple profiles (#82)
This PR does multiple things, which are:

1. Creates .databricks dir according to outcomes concluded in "bricks
configuration principles"
2. Puts the sync snapshots into a file whose names is tagged with
md5(concat(host, remote-path))
3. Saves both host and username in the bricks snapshot for debuggability

Tested manually:


https://user-images.githubusercontent.com/88374338/195672267-9dd90230-570f-49b7-847f-05a5a6fd8986.mov
2022-10-19 16:22:55 +02:00
Pieter Noordhuis 38a9dabcbe
Add command to make API calls (#80)
Not settled whether this should live as a top level command or hidden
under some debug scope. Either way, the ability to make arbitrary API
calls and leverage unified auth is a super useful tool.
2022-10-10 10:27:45 +02:00
shreyas-goenka ed56fc01cb
[DECO-68] Modify acceptance test to work with deco testing infra and represent vscode usecase (#78)
Contains changes to make this integration test work on our GitHub
actions testing env

1. use go run main.go to run bricks sync to run the latest bricks from
master
2. Log the output from the bricks sync process to allow for debugging
3. removed databricks.yml and instead rely on BRICKS_ROOT and other env
vars for auth and bricks sync
4. Added --persist-snapshot set to false to test full sync (same as is
used in the vscode extension

<img width="898" alt="Screenshot 2022-09-27 at 4 26 18 PM"
src="https://user-images.githubusercontent.com/88374338/192553769-7af08ca0-b73a-4cf6-a214-8c58edc4c3e5.png">

The additional logs in the picture above are from a wip PR in deco cli
that I made some changes to in order to make deco cli work with bricks :
https://github.com/databricks/eng-dev-ecosystem/pull/97
2022-10-05 13:28:53 +02:00
shreyas-goenka 0b754e6de8
[DECO-94] Execute uploads in parallel instead of sequentailly (#81)
Tested manually

Upload seems fast enough, delete API calls though have a much longer
turn around times

Additional optimizations that can be done if/when the need arises:
1. First time upload can be done using zip batching of the files


https://user-images.githubusercontent.com/88374338/192783332-9b2b19bc-d6c4-4a66-8dbc-e78287e6af1a.mov
2022-10-05 00:12:57 +02:00
Pieter Noordhuis a1b6fdb2e8
Update SDK (#79) 2022-09-27 09:58:55 -07:00
Serge Smertin f27cb7149d
Reduce nesting (#72) 2022-09-22 15:03:30 +02:00
Pieter Noordhuis 6258a1637d
Add environments to project configuration (#68) 2022-09-22 13:40:11 +02:00
shreyas-goenka 6bcb33bf07
Use env vars to read profile if databricks.yml is absent (#70)
Tested manually and by running sync_test.go integration test bricks sync
works
2022-09-21 13:55:37 +02:00
shreyas-goenka 731679cb4b
Add `persist-snapshot` to bricks sync (#66)
Tested manually

We are adding this flag because the default bricks sync is not robust
against changing the profile and other project config changes. This will
be used in the initial version of the vscode extention
2022-09-19 16:47:55 +02:00
Pieter Noordhuis 7cad8bda81
Respect project root in sync command (#63) 2022-09-16 15:18:46 +02:00
Pieter Noordhuis ec6c58f1d1
Fix relative path derivation if root path can be cleaned (#64) 2022-09-16 15:13:49 +02:00
Pieter Noordhuis a7701cc8f3
Store project object in context.Context instead of global (#61)
* Load project root from `BRICKS_ROOT` environment variable
* Rename project.Project -> project.Config
* Rename project.inner -> project.project
* Upgrade cobra to 1.5.0 for cmd.SetContext
2022-09-16 11:06:58 +02:00
shreyas-goenka 836ab58473
Fix flaky test `TestDiff` (#59) 2022-09-15 15:40:47 +02:00
Pieter Noordhuis 192a790155
Comment out flavor.go (#60)
The code here isn't used at the moment.
2022-09-15 09:46:11 +02:00
shreyas-goenka f9b66b3536
Make `bricks sync` feature work (#48)
Tested manually and partially by unit tests
2022-09-14 17:50:29 +02:00
Pieter Noordhuis 281f25a627
Don't depend on working directory in folders.FindDirWithLeaf (#54)
This is plumbing for allowing a configurable root that doesn't depend on
the working directory.
2022-09-14 15:08:55 +02:00
Pieter Noordhuis 14a45e701e
Run go mod tidy (#55) 2022-09-14 10:48:51 +02:00
Pieter Noordhuis 40194cc1a2
Scope snippets to Go (#51)
Confirmed they only show up in Go code now.
2022-09-13 17:58:59 +02:00
Fabian Jakobs a168eece47
Build 32 bit Windows (#47)
Win32 is needed by VSCode as a fallback.
2022-09-09 10:31:41 +02:00
shreyas-goenka 21bc774491
Replace scim Me terraform call with go sdk (#46)
This PR:
1. Replaces scim.Me call to use the go SDK instead of the terraform
client
2. Removes terraform client from bricks project

Tested manually that the scim.Me call works now and returns the correct
user
go build works
2022-09-08 15:50:00 +02:00
Fabian Jakobs d829ce36d5
fix release (#45) 2022-09-08 15:12:26 +02:00
Fabian Jakobs f309f68444
Fix release action and make sure to also build for Windows (#43) 2022-09-08 14:57:33 +02:00