cc5ed2bc76
fixed typo, and tested schema is working
2023-01-13 17:13:21 +01:00
926db54a6d
added tests for primitives and basic objects/arrays
2023-01-13 16:45:01 +01:00
987dd29357
added recursive limit
2023-01-13 12:09:19 +01:00
d46dbbce6a
WIP json schema support for bundles
2023-01-13 04:52:25 +01:00
3582037be6
Add nil check for retries.Info.Info ( #166 )
2023-01-12 18:58:36 +01:00
e48eb6ff50
Fix integration tests for windows ( #165 )
...
Successful run:
Azure:
https://github.com/databricks/eng-dev-ecosystem/actions/runs/3892131225
AWS:
https://github.com/databricks/eng-dev-ecosystem/actions/runs/3892432562
2023-01-11 13:32:02 +01:00
bf9f0c06b0
Fix sync integration test failing on aws-prod ( #164 )
...
The tests were failing because we were the list of expected files did
not include .gitignore
aws and azure test runs:
https://github.com/databricks/eng-dev-ecosystem/actions/runs/3891837322/jobs/6642552850
https://github.com/databricks/eng-dev-ecosystem/actions/runs/3891836306/jobs/6642550633
2023-01-11 11:50:27 +01:00
0d9ecb5643
Refactor and cover edge cases in sync integration tests ( #160 )
...
This PR:
1. Refactors the sync integration tests to make them more readable
2. Adds additional tests for edge cases we encountered during vscode
runs
3. Intensional side effect: sync integration tests are also green on
windows (see
https://github.com/databricks/eng-dev-ecosystem/actions/runs/3817365642/jobs/6493576727 )
Change in coverage
- We now test for python notebook <-> python file interconversion and
python notebook deletion being synced to workspace
- Tests are split up and are more focused on testing specific edge cases
2023-01-10 13:16:30 +01:00
b87b4b0f40
Added `bricks auth login` and `bricks auth token` ( #158 )
...
# Auth challenge (happy path)
Simplified description of [PKCE](https://oauth.net/2/pkce/ )
implementation:
```mermaid
sequenceDiagram
autonumber
actor User
User ->> CLI: type `bricks auth login HOST`
CLI ->>+ HOST: request OIDC endpoints
HOST ->>- CLI: auth & token endpoints
CLI ->> CLI: start embedded server to consume redirects (lock)
CLI -->>+ Auth Endpoint: open browser with RND1 + SHA256(RND2)
User ->>+ Auth Endpoint: Go through SSO
Auth Endpoint ->>- CLI: AUTH CODE + 'RND1 (redirect)
CLI ->>+ Token Endpoint: Exchange: AUTH CODE + RND2
Token Endpoint ->>- CLI: Access Token (JWT) + refresh + expiry
CLI ->> Token cache: Save Access Token (JWT) + refresh + expiry
CLI ->> User: success
```
# Token refresh (happy path)
```mermaid
sequenceDiagram
autonumber
actor User
User ->> CLI: type `bricks token HOST`
CLI ->> CLI: acquire lock (same local addr as redirect server)
CLI ->>+ Token cache: read token
critical token not expired
Token cache ->>- User: JWT (without refresh)
option token is expired
CLI ->>+ HOST: request OIDC endpoints
HOST ->>- CLI: auth & token endpoints
CLI ->>+ Token Endpoint: refresh token
Token Endpoint ->>- CLI: JWT (refreshed)
CLI ->> Token cache: save JWT (refreshed)
CLI ->> User: JWT (refreshed)
option no auth for host
CLI -X User: no auth configured
end
```
2023-01-06 16:15:57 +01:00
a59136f77f
Use []byte for files in workspace ( #162 )
2023-01-05 12:03:31 +01:00
a9b82aa1c7
Remove version suffix from snapshot binaries ( #159 )
...
After this change it's possible to have a stable symlink for
development:
<img width="172" alt="image"
src="https://user-images.githubusercontent.com/259697/209996850-68bcb6da-b8f9-4b5a-9ac3-ff3ef35cd751.png ">
2023-01-03 12:15:21 +01:00
6d04eb0f0c
Update to Go SDK v0.2.0 ( #157 )
...
This PR updates Go SDK to v0.2.0
2022-12-28 11:32:04 +01:00
8f4461904b
Define flags for running jobs and pipelines ( #146 )
2022-12-23 15:17:16 +01:00
49aa858b89
Run command must always take a single argument ( #156 )
2022-12-22 16:19:38 +01:00
61ef0ba8c6
Handle nil environment ( #154 )
2022-12-22 15:31:32 +01:00
198eefcf39
Fix folder syncing on windows ( #149 )
...
Tested by running the unit and integration tests locally
Tested manually on windows
Screenshot from windows sync logs indicating that the correct slashed
for paths were used:
<img width="623" alt="Screenshot 2022-12-21 at 9 09 13 PM"
src="https://user-images.githubusercontent.com/88374338/208943937-146670b2-1afd-4e0b-8f4e-6091c8c7e17a.png ">
@pietern with this the state machine for syncing becomes slightly more
complicated, indicating a stronger need for a tree based approach herre
Co-authored-by: Pieter Noordhuis <pieter.noordhuis@databricks.com>
2022-12-22 11:35:32 +01:00
f3112d90ad
Work on release actions ( #153 )
...
* Include Go version in cache key
* Store build artifacts by operating systems
2022-12-22 11:01:50 +01:00
f5cb9e06a9
Add workflow for building release snapshots ( #152 )
...
Use custom caching because goreleaser performs a cross platform build
and therefore we cannot share a cache with the test builds.
2022-12-22 10:25:24 +01:00
7f83463ca3
Bump SDK to latest ( #151 )
2022-12-22 09:46:17 +01:00
ce55284398
Run go mod tidy in fmt job ( #150 )
2022-12-22 09:33:46 +01:00
fdb8c97f6b
Exit with non-zero status on errors ( #148 )
2022-12-21 11:58:51 +01:00
a354fa1f77
Only display usage string on flag errors ( #147 )
2022-12-21 11:38:30 +01:00
f70a5814f6
Run initialize phase from validate command ( #145 )
...
This lets the user know if their workspace connection is setup
correctly.
2022-12-16 15:00:48 +01:00
4026b2cda2
Mutator to convert paths to local notebooks files into artifacts ( #144 )
...
This lets you write:
```yaml
libraries:
- notebook:
path: ./events.sql
```
Instead of:
```yaml
artifacts:
events_sql:
notebook:
path: ./events.sql
libraries:
- notebook:
path: "${artifacts.events_sql.notebook.remote_path}"
```
2022-12-16 14:49:23 +01:00
1a9a431b97
No need for nil check on map ( #143 )
2022-12-15 21:28:27 +01:00
24a3b90713
Add "default" flag to environment block ( #142 )
...
If the environment is not set through command line argument or
environment variable, the bundle loads either 1) the only environment,
2) the only environment with the default flag set.
2022-12-15 21:28:14 +01:00
35243db33c
Automatically install Terraform if needed ( #141 )
...
Users can opt out and use the system-installed version with the
following configuration:
```
bundle:
terraform:
exec_path: terraform
```
This will find the binary in $PATH and replace it with the found value.
If this is not set, the initialize phase will install Terraform in the
bundle's cache directory.
2022-12-15 17:30:33 +01:00
32a37c1b83
Use filer.Filer in bundle/deployer/locker ( #136 )
...
Summary:
* All remote path arguments for deployer and locker are now relative to
root specified at initialization
* The workspace client is now a struct field so it doesn't have to be
passed around
2022-12-15 17:16:07 +01:00
b111416fe5
Add `bricks bundle run` command ( #134 )
2022-12-15 15:12:47 +01:00
72e89bf33c
Use pointers to resources in bundle configuration ( #140 )
...
Avoid copy-by-value when iterating over these maps.
2022-12-15 13:00:41 +01:00
4e834857e6
Extract filer path handling into separate type ( #138 )
...
This makes it reusable for the DBFS filer.
2022-12-14 23:41:37 +01:00
d0bd74c116
Run Go formatting with 1.19 ( #137 )
...
See https://tip.golang.org/doc/go1.19#go-doc .
2022-12-14 15:59:47 +01:00
12aae35519
Abstract over file handling with WSFS or DBFS through filer interface ( #135 )
2022-12-14 15:37:14 +01:00
fa458406ea
Bump github.com/databricks/databricks-sdk-go from 0.1.0 to 0.1.1 ( #121 )
2022-12-14 11:21:44 +01:00
4631c70279
Bump github.com/spf13/cobra from 1.5.0 to 1.6.1 ( #115 )
2022-12-14 11:00:44 +01:00
7f3094e4d7
Bump golang.org/x/mod from 0.6.0 to 0.7.0 ( #114 )
2022-12-14 10:58:30 +01:00
58e6ba3483
Work on GitHub actions ( #133 )
...
This does:
* Use actions/checkout@v3 (fixes node.js v12 deprecation warning)
* Pin Go version to 1.18.8 to make caching work better
* Remove checkout of submodules (we don't have any anymore)
2022-12-12 16:51:27 +01:00
d713521d63
Convert job task libraries to TF JSON ( #132 )
2022-12-12 16:36:59 +01:00
b42768801d
[DECO-396] Post mortem followups on sync deletes repo ( #119 )
...
This PR:
- Implements safeguards for not accidentally/maliciously deleting repos
by sanitizing relative paths
- Adds versioning for snapshot schemas to allow invalidation if needed
- Adds logic to delete preexisting remote artifacts that might not have
been cleaned up properly if they conflict with an upload
- A bunch of tests for the changes here
Co-authored-by: Pieter Noordhuis <pieter.noordhuis@databricks.com>
2022-12-12 14:31:06 +01:00
c255bd686a
Define deploy command as sequence of build phases ( #129 )
2022-12-12 12:49:25 +01:00
8640696b4b
Add minimal test for conversion to TF JSON format ( #130 )
2022-12-12 11:31:28 +01:00
94a86972e5
Allow multiple lookup functions for interpolation ( #128 )
2022-12-12 10:48:52 +01:00
3f8e233a18
Function to limit interpolation to specific path ( #127 )
...
New function `IncludeLookupsInPath` is counterpart to
`ExcludeLookupsInPath`.
2022-12-12 10:30:17 +01:00
cb16ad1184
Add RequireSuccessfulRun helper ( #125 )
2022-12-09 15:41:04 +01:00
94f884f0a7
Run bricks sync in-process instead of through go run ( #123 )
2022-12-09 11:47:06 +01:00
4f668fc58b
Mutators to work with Terraform ( #124 )
...
This includes 3 mutators:
* Interpolate resources references to TF compatible format
* Convert resources struct to TF JSON format and write it to disk
* Run TF apply
2022-12-09 08:57:30 +01:00
ff89c9d06f
Generate equivalent Go types from Terraform provider schema ( #122 )
...
It contains:
* `codegen` -- this turns the schema of the Databricks Terraform provider into Go types.
* `schema` -- the output of the above.
2022-12-06 16:26:19 +01:00
d9d295f2a9
Implement Terraform state synchronization and deploy ( #98 )
...
https://user-images.githubusercontent.com/88374338/203669797-abebf99e-8fa6-4d6e-b57a-abd172d8020d.mov
2022-12-06 00:40:45 +01:00
f3ac75ab4d
Remove .gitmodules ( #120 )
...
No longer needed since #110 .
2022-12-05 09:55:13 +01:00
d5474c9673
Revert "Rename jobs -> workflows" ( #118 )
...
This reverts PR #111 .
This reverts commit 230811031f
2022-12-01 22:39:15 +01:00
Shreyas Goenka
Pieter Noordhuis
Kartik Gupta
shreyas-goenka
Serge Smertin
dependabot[bot]
