Commit Graph

446 Commits

Author SHA1 Message Date
shreyas-goenka 0d9ecb5643
Refactor and cover edge cases in sync integration tests (#160)
This PR:
1. Refactors the sync integration tests to make them more readable
2. Adds additional tests for edge cases we encountered during vscode
runs
3. Intensional side effect: sync integration tests are also green on
windows (see
https://github.com/databricks/eng-dev-ecosystem/actions/runs/3817365642/jobs/6493576727)

Change in coverage

- We now test for python notebook <-> python file interconversion and
python notebook deletion being synced to workspace
- Tests are split up and are more focused on testing specific edge cases
2023-01-10 13:16:30 +01:00
Serge Smertin b87b4b0f40
Added `bricks auth login` and `bricks auth token` (#158)
# Auth challenge (happy path)

Simplified description of [PKCE](https://oauth.net/2/pkce/)
implementation:

```mermaid
sequenceDiagram
    autonumber
    actor User
    
    User ->> CLI: type `bricks auth login HOST`
    CLI ->>+ HOST: request OIDC endpoints
    HOST ->>- CLI: auth & token endpoints
    CLI ->> CLI: start embedded server to consume redirects (lock)
    CLI -->>+ Auth Endpoint: open browser with RND1 + SHA256(RND2)

    User ->>+ Auth Endpoint: Go through SSO
    Auth Endpoint ->>- CLI: AUTH CODE + 'RND1 (redirect)

    CLI ->>+ Token Endpoint: Exchange: AUTH CODE + RND2
    Token Endpoint ->>- CLI: Access Token (JWT) + refresh + expiry
    CLI ->> Token cache: Save Access Token (JWT) + refresh + expiry
    CLI ->> User: success
```

# Token refresh (happy path)

```mermaid
sequenceDiagram
    autonumber
    actor User
    
    User ->> CLI: type `bricks token HOST`
    
    CLI ->> CLI: acquire lock (same local addr as redirect server)
    CLI ->>+ Token cache: read token

    critical token not expired
    Token cache ->>- User: JWT (without refresh)

    option token is expired
    CLI ->>+ HOST: request OIDC endpoints
    HOST ->>- CLI: auth & token endpoints
    CLI ->>+ Token Endpoint: refresh token
    Token Endpoint ->>- CLI: JWT (refreshed)
    CLI ->> Token cache: save JWT (refreshed)
    CLI ->> User: JWT (refreshed)
    
    option no auth for host
    CLI -X User: no auth configured
    end
```
2023-01-06 16:15:57 +01:00
Pieter Noordhuis a59136f77f
Use []byte for files in workspace (#162) 2023-01-05 12:03:31 +01:00
Serge Smertin a9b82aa1c7
Remove version suffix from snapshot binaries (#159)
After this change it's possible to have a stable symlink for
development:

<img width="172" alt="image"
src="https://user-images.githubusercontent.com/259697/209996850-68bcb6da-b8f9-4b5a-9ac3-ff3ef35cd751.png">
2023-01-03 12:15:21 +01:00
Serge Smertin 6d04eb0f0c
Update to Go SDK v0.2.0 (#157)
This PR updates Go SDK to v0.2.0
2022-12-28 11:32:04 +01:00
Pieter Noordhuis 8f4461904b
Define flags for running jobs and pipelines (#146) 2022-12-23 15:17:16 +01:00
Pieter Noordhuis 49aa858b89
Run command must always take a single argument (#156) 2022-12-22 16:19:38 +01:00
Pieter Noordhuis 61ef0ba8c6
Handle nil environment (#154) 2022-12-22 15:31:32 +01:00
shreyas-goenka 198eefcf39
Fix folder syncing on windows (#149)
Tested by running the unit and integration tests locally

Tested manually on windows

Screenshot from windows sync logs indicating that the correct slashed
for paths were used:
<img width="623" alt="Screenshot 2022-12-21 at 9 09 13 PM"
src="https://user-images.githubusercontent.com/88374338/208943937-146670b2-1afd-4e0b-8f4e-6091c8c7e17a.png">

@pietern with this the state machine for syncing becomes slightly more
complicated, indicating a stronger need for a tree based approach herre

Co-authored-by: Pieter Noordhuis <pieter.noordhuis@databricks.com>
2022-12-22 11:35:32 +01:00
Pieter Noordhuis f3112d90ad
Work on release actions (#153)
* Include Go version in cache key
* Store build artifacts by operating systems
2022-12-22 11:01:50 +01:00
Pieter Noordhuis f5cb9e06a9
Add workflow for building release snapshots (#152)
Use custom caching because goreleaser performs a cross platform build
and therefore we cannot share a cache with the test builds.
2022-12-22 10:25:24 +01:00
Pieter Noordhuis 7f83463ca3
Bump SDK to latest (#151) 2022-12-22 09:46:17 +01:00
Pieter Noordhuis ce55284398
Run go mod tidy in fmt job (#150) 2022-12-22 09:33:46 +01:00
Pieter Noordhuis fdb8c97f6b
Exit with non-zero status on errors (#148) 2022-12-21 11:58:51 +01:00
Pieter Noordhuis a354fa1f77
Only display usage string on flag errors (#147) 2022-12-21 11:38:30 +01:00
Pieter Noordhuis f70a5814f6
Run initialize phase from validate command (#145)
This lets the user know if their workspace connection is setup
correctly.
2022-12-16 15:00:48 +01:00
Pieter Noordhuis 4026b2cda2
Mutator to convert paths to local notebooks files into artifacts (#144)
This lets you write:
```yaml
libraries:
  - notebook:
      path: ./events.sql
```

Instead of:
```yaml
artifacts:
  events_sql:
    notebook:
      path: ./events.sql

libraries:
  - notebook:
      path: "${artifacts.events_sql.notebook.remote_path}"
```
2022-12-16 14:49:23 +01:00
Pieter Noordhuis 1a9a431b97
No need for nil check on map (#143) 2022-12-15 21:28:27 +01:00
Pieter Noordhuis 24a3b90713
Add "default" flag to environment block (#142)
If the environment is not set through command line argument or
environment variable, the bundle loads either 1) the only environment,
2) the only environment with the default flag set.
2022-12-15 21:28:14 +01:00
Pieter Noordhuis 35243db33c
Automatically install Terraform if needed (#141)
Users can opt out and use the system-installed version with the
following configuration:

```
bundle:
  terraform:
    exec_path: terraform
```

This will find the binary in $PATH and replace it with the found value.

If this is not set, the initialize phase will install Terraform in the
bundle's cache directory.
2022-12-15 17:30:33 +01:00
Pieter Noordhuis 32a37c1b83
Use filer.Filer in bundle/deployer/locker (#136)
Summary:
* All remote path arguments for deployer and locker are now relative to
root specified at initialization
* The workspace client is now a struct field so it doesn't have to be
passed around
2022-12-15 17:16:07 +01:00
Pieter Noordhuis b111416fe5
Add `bricks bundle run` command (#134) 2022-12-15 15:12:47 +01:00
Pieter Noordhuis 72e89bf33c
Use pointers to resources in bundle configuration (#140)
Avoid copy-by-value when iterating over these maps.
2022-12-15 13:00:41 +01:00
Pieter Noordhuis 4e834857e6
Extract filer path handling into separate type (#138)
This makes it reusable for the DBFS filer.
2022-12-14 23:41:37 +01:00
Pieter Noordhuis d0bd74c116
Run Go formatting with 1.19 (#137)
See https://tip.golang.org/doc/go1.19#go-doc.
2022-12-14 15:59:47 +01:00
Pieter Noordhuis 12aae35519
Abstract over file handling with WSFS or DBFS through filer interface (#135) 2022-12-14 15:37:14 +01:00
dependabot[bot] fa458406ea
Bump github.com/databricks/databricks-sdk-go from 0.1.0 to 0.1.1 (#121) 2022-12-14 11:21:44 +01:00
dependabot[bot] 4631c70279
Bump github.com/spf13/cobra from 1.5.0 to 1.6.1 (#115) 2022-12-14 11:00:44 +01:00
dependabot[bot] 7f3094e4d7
Bump golang.org/x/mod from 0.6.0 to 0.7.0 (#114) 2022-12-14 10:58:30 +01:00
Pieter Noordhuis 58e6ba3483
Work on GitHub actions (#133)
This does:
* Use actions/checkout@v3 (fixes node.js v12 deprecation warning)
* Pin Go version to 1.18.8 to make caching work better
* Remove checkout of submodules (we don't have any anymore)
2022-12-12 16:51:27 +01:00
Pieter Noordhuis d713521d63
Convert job task libraries to TF JSON (#132) 2022-12-12 16:36:59 +01:00
shreyas-goenka b42768801d
[DECO-396] Post mortem followups on sync deletes repo (#119)
This PR:
- Implements safeguards for not accidentally/maliciously deleting repos
by sanitizing relative paths
- Adds versioning for snapshot schemas to allow invalidation if needed
- Adds logic to delete preexisting remote artifacts that might not have
been cleaned up properly if they conflict with an upload
- A bunch of tests for the changes here

Co-authored-by: Pieter Noordhuis <pieter.noordhuis@databricks.com>
2022-12-12 14:31:06 +01:00
Pieter Noordhuis c255bd686a
Define deploy command as sequence of build phases (#129) 2022-12-12 12:49:25 +01:00
Pieter Noordhuis 8640696b4b
Add minimal test for conversion to TF JSON format (#130) 2022-12-12 11:31:28 +01:00
Pieter Noordhuis 94a86972e5
Allow multiple lookup functions for interpolation (#128) 2022-12-12 10:48:52 +01:00
Pieter Noordhuis 3f8e233a18
Function to limit interpolation to specific path (#127)
New function `IncludeLookupsInPath` is counterpart to
`ExcludeLookupsInPath`.
2022-12-12 10:30:17 +01:00
Pieter Noordhuis cb16ad1184
Add RequireSuccessfulRun helper (#125) 2022-12-09 15:41:04 +01:00
Pieter Noordhuis 94f884f0a7
Run bricks sync in-process instead of through go run (#123) 2022-12-09 11:47:06 +01:00
Pieter Noordhuis 4f668fc58b
Mutators to work with Terraform (#124)
This includes 3 mutators:
* Interpolate resources references to TF compatible format
* Convert resources struct to TF JSON format and write it to disk
* Run TF apply
2022-12-09 08:57:30 +01:00
Pieter Noordhuis ff89c9d06f
Generate equivalent Go types from Terraform provider schema (#122)
It contains:
* `codegen` -- this turns the schema of the Databricks Terraform provider into Go types.
* `schema` -- the output of the above.
2022-12-06 16:26:19 +01:00
shreyas-goenka d9d295f2a9
Implement Terraform state synchronization and deploy (#98)
https://user-images.githubusercontent.com/88374338/203669797-abebf99e-8fa6-4d6e-b57a-abd172d8020d.mov
2022-12-06 00:40:45 +01:00
Pieter Noordhuis f3ac75ab4d
Remove .gitmodules (#120)
No longer needed since #110.
2022-12-05 09:55:13 +01:00
Pieter Noordhuis d5474c9673
Revert "Rename jobs -> workflows" (#118)
This reverts PR #111.

This reverts commit 230811031f.
2022-12-01 22:39:15 +01:00
Pieter Noordhuis cdc776d89e
Parameterize interpolation function (#117)
By specifying a function typed `LookupFunction` the caller can customize
which path expressions to interpolate and which ones to skip. When we
express dependencies between resources their values are known by
Terraform at deploy time. Therefore, we have to skip interpolation for
`${resources.jobs.my_job.id}` and instead rewrite it to
`${databricks_job.my_job.id}` before passing it along to Terraform.
2022-12-01 22:38:49 +01:00
Serge Smertin 487bf6fd5c
Use Databricks Go SDK v0.1.0 (#110)
This PR pins the version of Databricks SDK for Go to v0.1.0
2022-12-01 12:17:36 +01:00
Pieter Noordhuis 34af98a8c3
Mutators to define current user and default artifact path (#112) 2022-12-01 11:17:29 +01:00
Pieter Noordhuis 230811031f
Rename jobs -> workflows (#111) 2022-12-01 09:35:21 +01:00
Pieter Noordhuis c4d63eac70
Rudimentary interpolation support (#108)
Performs interpolation on string field.

It looks for patterns `${foo.bar}` where `foo.bar` points to a string
field in the configuration data model.

It does not support traversal (e.g. `${foo}` with `foo` equal
to`${bar}`), hence "rudimentary".
2022-12-01 09:33:42 +01:00
Pieter Noordhuis 4064a21797
Function to return bundle's cache directory (#109)
Parallel of `project.CacheDir()` introduced in
https://github.com/databricks/bricks/pull/82.
2022-11-30 14:40:41 +01:00
Pieter Noordhuis e1669b0352
Model code artifacts (#107)
This adds:
* Top level "artifacts" configuration key
* Support for notebooks (does language detection and upload)
* Merge of per-environment artifacts (or artifact overrides) into top level
2022-11-30 14:15:22 +01:00