Commit Graph

1280 Commits

Author SHA1 Message Date
Pieter Noordhuis 4026b2cda2
Mutator to convert paths to local notebooks files into artifacts (#144)
This lets you write:
```yaml
libraries:
  - notebook:
      path: ./events.sql
```

Instead of:
```yaml
artifacts:
  events_sql:
    notebook:
      path: ./events.sql

libraries:
  - notebook:
      path: "${artifacts.events_sql.notebook.remote_path}"
```
2022-12-16 14:49:23 +01:00
Pieter Noordhuis 1a9a431b97
No need for nil check on map (#143) 2022-12-15 21:28:27 +01:00
Pieter Noordhuis 24a3b90713
Add "default" flag to environment block (#142)
If the environment is not set through command line argument or
environment variable, the bundle loads either 1) the only environment,
2) the only environment with the default flag set.
2022-12-15 21:28:14 +01:00
Pieter Noordhuis 35243db33c
Automatically install Terraform if needed (#141)
Users can opt out and use the system-installed version with the
following configuration:

```
bundle:
  terraform:
    exec_path: terraform
```

This will find the binary in $PATH and replace it with the found value.

If this is not set, the initialize phase will install Terraform in the
bundle's cache directory.
2022-12-15 17:30:33 +01:00
Pieter Noordhuis 32a37c1b83
Use filer.Filer in bundle/deployer/locker (#136)
Summary:
* All remote path arguments for deployer and locker are now relative to
root specified at initialization
* The workspace client is now a struct field so it doesn't have to be
passed around
2022-12-15 17:16:07 +01:00
Pieter Noordhuis b111416fe5
Add `bricks bundle run` command (#134) 2022-12-15 15:12:47 +01:00
Pieter Noordhuis 72e89bf33c
Use pointers to resources in bundle configuration (#140)
Avoid copy-by-value when iterating over these maps.
2022-12-15 13:00:41 +01:00
Pieter Noordhuis 4e834857e6
Extract filer path handling into separate type (#138)
This makes it reusable for the DBFS filer.
2022-12-14 23:41:37 +01:00
Pieter Noordhuis d0bd74c116
Run Go formatting with 1.19 (#137)
See https://tip.golang.org/doc/go1.19#go-doc.
2022-12-14 15:59:47 +01:00
Pieter Noordhuis 12aae35519
Abstract over file handling with WSFS or DBFS through filer interface (#135) 2022-12-14 15:37:14 +01:00
dependabot[bot] fa458406ea
Bump github.com/databricks/databricks-sdk-go from 0.1.0 to 0.1.1 (#121) 2022-12-14 11:21:44 +01:00
dependabot[bot] 4631c70279
Bump github.com/spf13/cobra from 1.5.0 to 1.6.1 (#115) 2022-12-14 11:00:44 +01:00
dependabot[bot] 7f3094e4d7
Bump golang.org/x/mod from 0.6.0 to 0.7.0 (#114) 2022-12-14 10:58:30 +01:00
Pieter Noordhuis 58e6ba3483
Work on GitHub actions (#133)
This does:
* Use actions/checkout@v3 (fixes node.js v12 deprecation warning)
* Pin Go version to 1.18.8 to make caching work better
* Remove checkout of submodules (we don't have any anymore)
2022-12-12 16:51:27 +01:00
Pieter Noordhuis d713521d63
Convert job task libraries to TF JSON (#132) 2022-12-12 16:36:59 +01:00
shreyas-goenka b42768801d
[DECO-396] Post mortem followups on sync deletes repo (#119)
This PR:
- Implements safeguards for not accidentally/maliciously deleting repos
by sanitizing relative paths
- Adds versioning for snapshot schemas to allow invalidation if needed
- Adds logic to delete preexisting remote artifacts that might not have
been cleaned up properly if they conflict with an upload
- A bunch of tests for the changes here

Co-authored-by: Pieter Noordhuis <pieter.noordhuis@databricks.com>
2022-12-12 14:31:06 +01:00
Pieter Noordhuis c255bd686a
Define deploy command as sequence of build phases (#129) 2022-12-12 12:49:25 +01:00
Pieter Noordhuis 8640696b4b
Add minimal test for conversion to TF JSON format (#130) 2022-12-12 11:31:28 +01:00
Pieter Noordhuis 94a86972e5
Allow multiple lookup functions for interpolation (#128) 2022-12-12 10:48:52 +01:00
Pieter Noordhuis 3f8e233a18
Function to limit interpolation to specific path (#127)
New function `IncludeLookupsInPath` is counterpart to
`ExcludeLookupsInPath`.
2022-12-12 10:30:17 +01:00
Pieter Noordhuis cb16ad1184
Add RequireSuccessfulRun helper (#125) 2022-12-09 15:41:04 +01:00
Pieter Noordhuis 94f884f0a7
Run bricks sync in-process instead of through go run (#123) 2022-12-09 11:47:06 +01:00
Pieter Noordhuis 4f668fc58b
Mutators to work with Terraform (#124)
This includes 3 mutators:
* Interpolate resources references to TF compatible format
* Convert resources struct to TF JSON format and write it to disk
* Run TF apply
2022-12-09 08:57:30 +01:00
Pieter Noordhuis ff89c9d06f
Generate equivalent Go types from Terraform provider schema (#122)
It contains:
* `codegen` -- this turns the schema of the Databricks Terraform provider into Go types.
* `schema` -- the output of the above.
2022-12-06 16:26:19 +01:00
shreyas-goenka d9d295f2a9
Implement Terraform state synchronization and deploy (#98)
https://user-images.githubusercontent.com/88374338/203669797-abebf99e-8fa6-4d6e-b57a-abd172d8020d.mov
2022-12-06 00:40:45 +01:00
Pieter Noordhuis f3ac75ab4d
Remove .gitmodules (#120)
No longer needed since #110.
2022-12-05 09:55:13 +01:00
Pieter Noordhuis d5474c9673
Revert "Rename jobs -> workflows" (#118)
This reverts PR #111.

This reverts commit 230811031f.
2022-12-01 22:39:15 +01:00
Pieter Noordhuis cdc776d89e
Parameterize interpolation function (#117)
By specifying a function typed `LookupFunction` the caller can customize
which path expressions to interpolate and which ones to skip. When we
express dependencies between resources their values are known by
Terraform at deploy time. Therefore, we have to skip interpolation for
`${resources.jobs.my_job.id}` and instead rewrite it to
`${databricks_job.my_job.id}` before passing it along to Terraform.
2022-12-01 22:38:49 +01:00
Serge Smertin 487bf6fd5c
Use Databricks Go SDK v0.1.0 (#110)
This PR pins the version of Databricks SDK for Go to v0.1.0
2022-12-01 12:17:36 +01:00
Pieter Noordhuis 34af98a8c3
Mutators to define current user and default artifact path (#112) 2022-12-01 11:17:29 +01:00
Pieter Noordhuis 230811031f
Rename jobs -> workflows (#111) 2022-12-01 09:35:21 +01:00
Pieter Noordhuis c4d63eac70
Rudimentary interpolation support (#108)
Performs interpolation on string field.

It looks for patterns `${foo.bar}` where `foo.bar` points to a string
field in the configuration data model.

It does not support traversal (e.g. `${foo}` with `foo` equal
to`${bar}`), hence "rudimentary".
2022-12-01 09:33:42 +01:00
Pieter Noordhuis 4064a21797
Function to return bundle's cache directory (#109)
Parallel of `project.CacheDir()` introduced in
https://github.com/databricks/bricks/pull/82.
2022-11-30 14:40:41 +01:00
Pieter Noordhuis e1669b0352
Model code artifacts (#107)
This adds:
* Top level "artifacts" configuration key
* Support for notebooks (does language detection and upload)
* Merge of per-environment artifacts (or artifact overrides) into top level
2022-11-30 14:15:22 +01:00
shreyas-goenka c6b3b35e98
[DECO-396] Send delete file requests with recursive set to false (#106)
Safeguard so bugs do not delete large amount of remote files
2022-11-30 13:56:52 +01:00
shreyas-goenka 2ebfa5f369
Run unit tests on windows and macos (#103)
Unit tests are now run in all three big OS. 

Some of the changes are to make the tests green for windows while we are
skipping some of the other tests on windows/macOS to make the tests
pass. This is a temporary measure and we will incrementally migrate
these tests over so there is parity in unit testing along all three
environments!
2022-11-28 11:34:25 +01:00
Pieter Noordhuis b88b35a510
Move mutator interface to top level bundle package (#105)
While working on artifact upload and workspace interrogation I realized
this mutator interface needs to:
1. Operate at the whole bundle level so it can apply to both
configuration and internal state
2. Include a `context.Context` parameter for a) long running operations
and b) progress reporting

Previous interface:
```
Apply(*config.Root) ([]Mutator, error)
```

New interface:
```
Apply(context.Context, *Bundle) ([]Mutator, error)
```
2022-11-28 10:59:43 +01:00
Pieter Noordhuis 5c916a6fb4
Store specified environment in configuration for reference (#104) 2022-11-28 10:10:13 +01:00
shreyas-goenka 761de12a0d
Fix incorrect remote path arg to APIs in windows on bricks sync (#101)
Integration tests testing sync on ubuntu are green:
<img width="794" alt="Screenshot 2022-11-25 at 2 05 11 PM"
src="https://user-images.githubusercontent.com/88374338/203991936-f0478c35-5d97-429b-b373-f050d9a90cff.png">

@fjakobs tested this on windows manually
2022-11-25 14:13:15 +01:00
Pieter Noordhuis 8e786d76a9
Update databricks-sdk-go to latest (#102) 2022-11-24 21:41:57 +01:00
Pieter Noordhuis 07f07694a4
Function to return workspace client on bundle.Bundle (#100)
Complementary command to check the identity in the context of a bundle
environment:

For example:
```
bricks bundle debug whoami -e development
```
2022-11-23 15:20:03 +01:00
shreyas-goenka 696e337516
Update go sdk version (#97) 2022-11-22 14:52:45 +01:00
shreyas-goenka b25bd318e3
Deal with notebook remote filenames stripping .py (#90)
This PR introduces tracking of remote names and local names of files in snapshots to disambiguate between files which might have the same remote name and handle clean deleting of files whose remote name changes due (eg. python notebook getting converted to a python notebook)
2022-11-21 23:42:09 +01:00
Pieter Noordhuis ab1df558a2
Test that YAML anchors work (#96) 2022-11-21 15:40:27 +01:00
Pieter Noordhuis 3b351d3b00
Add command that writes the materialized bundle configuration to stdout (#95)
Used to inspect the bundle configuration after loading and merging all
files.

Once we add variable interpolation this command could show the result
after interpolation as well.

Each of the mutations to this configuration is observable, so we could
add a mode that writes each of the intermediate versions to disk for
even more fine grained introspection.
2022-11-21 15:39:53 +01:00
Pieter Noordhuis 195eb7f0f9
Add job and pipeline structs (#94) 2022-11-18 11:12:24 +01:00
Pieter Noordhuis e47fa61951
Skeleton for configuration loading and mutation (#92)
Load a tree of configuration files anchored at `bundle.yml` into the
`config.Root` struct.

All mutations (from setting defaults to merging files) are observable
through the `mutator.Mutator` interface.
2022-11-18 10:57:31 +01:00
Pieter Noordhuis 6a8c9f22b8
Update databricks-sdk-go to latest (#93) 2022-11-17 11:24:09 +01:00
Shreyas Goenka 0c24e6f82e
Revert "WIP initial version of the workspace file lock done"
This reverts commit 02eec1f990.
2022-11-16 23:50:17 +01:00
Shreyas Goenka 02eec1f990
WIP initial version of the workspace file lock done 2022-11-16 17:30:46 +01:00