Commit Graph

355 Commits

Author SHA1 Message Date
Andrew Nester 3c4d6f637f
Changed service template to correctly handle required positional arguments (#405) 2023-05-26 14:46:08 +02:00
Andrew Nester 6141476ca2
Added support for bundle.Seq, simplified Mutator.Apply interface (#403)
## Changes
Added support for `bundle.Seq`, simplified `Mutator.Apply` interface by
removing list of mutators from return values/

## Tests
1. Ran `cli bundle deploy` and interrupted it with Cmd + C mid execution
so lock is not released
2. Ran `cli bundle deploy` top make sure that CLI is not trying to
release lock when it fail to acquire it
```
andrew.nester@HFW9Y94129 multiples-tasks % cli bundle deploy
Starting upload of bundle files
Uploaded bundle files at /Users/andrew.nester@databricks.com/.bundle/simple-task/development/files!

^C
andrew.nester@HFW9Y94129 multiples-tasks % cli bundle deploy
Error: deploy lock acquired by andrew.nester@databricks.com at 2023-05-24 12:10:23.050343 +0200 CEST. Use --force to override
```
2023-05-24 14:45:19 +02:00
Pieter Noordhuis 46df551816
Update to Go SDK v0.9.0 (#396)
## Changes

See https://github.com/databricks/databricks-sdk-go/releases/tag/v0.9.0.

## Tests

Ran integration tests manually.
2023-05-22 21:27:22 +02:00
Pieter Noordhuis d86a1f0847
Add version flag to print version and exit (#394)
## Changes

With this PR, all of the command below print version and exit:
```
$ databricks -v
Databricks CLI v0.100.1-dev+4d3fa76
$ databricks --version
Databricks CLI v0.100.1-dev+4d3fa76
$ databricks version  
Databricks CLI v0.100.1-dev+4d3fa76
```

## Tests

Added integration test for each flag or command.
2023-05-22 20:55:42 +02:00
Fabian Jakobs 055e528173
Rename: bricks -> databricks (#393)
## Changes
related to https://github.com/databricks/databricks-vscode/pull/721

## Rename env vars

`BRICKS_CLI_PATH` -> `DATABRICKS_CLI_PATH`
`BRICKS_OUTPUT_FORMAT` -> `DATABRICKS_OUTPUT_FORMAT`
`BRICKS_LOG_FILE` -> `DATABRICKS_LOG_FILE`
`BRICKS_LOG_LEVEL` -> `DATABRICKS_LOG_LEVEL`
`BRICKS_LOG_FORMAT` -> `DATABRICKS_LOG_FORMAT`
`BRICKS_PROGRESS_FORMAT` -> `DATABRICKS_CLI_PROGRESS_FORMAT`
`BRICKS_UPSTREAM` -> `DATABRICKS_CLI_UPSTREAM`
`BRICKS_UPSTREAM_VERSION` -> `DATABRICKS_CLI_UPSTREAM_VERSION`
2023-05-22 16:40:50 +02:00
Pieter Noordhuis 98ebb78c9b
Rename bricks -> databricks (#389)
## Changes

Rename all instances of "bricks" to "databricks".

## Tests

* Confirmed the goreleaser build works, uses the correct new binary
name, and produces the right archives.
* Help output is confirmed to be correct.
* Output of `git grep -w bricks` is minimal with a couple changes
remaining for after the repository rename.
2023-05-16 18:35:39 +02:00
shreyas-goenka c5e940f664
Add support for variables in bundle config (#359)
## Changes
This PR now allows you to define variables in the bundle config and set
them in three ways
1. command line args
2. process environment variable
3. in the bundle config itself

## Tests
manually, unit, and black box tests

---------

Co-authored-by: Miles Yucht <miles@databricks.com>
2023-05-15 11:34:05 +02:00
Serge Smertin 4c4a293015
Added OpenAPI command coverage (#357)
This PR adds the following command groups:

## Workspace-level command groups

 * `bricks alerts` - The alerts API can be used to perform CRUD operations on alerts.
 * `bricks catalogs` - A catalog is the first layer of Unity Catalog’s three-level namespace.
 * `bricks cluster-policies` - Cluster policy limits the ability to configure clusters based on a set of rules.
 * `bricks clusters` - The Clusters API allows you to create, start, edit, list, terminate, and delete clusters.
 * `bricks current-user` - This API allows retrieving information about currently authenticated user or service principal.
 * `bricks dashboards` - In general, there is little need to modify dashboards using the API.
 * `bricks data-sources` - This API is provided to assist you in making new query objects.
 * `bricks experiments` - MLflow Experiment tracking.
 * `bricks external-locations` - An external location is an object that combines a cloud storage path with a storage credential that authorizes access to the cloud storage path.
 * `bricks functions` - Functions implement User-Defined Functions (UDFs) in Unity Catalog.
 * `bricks git-credentials` - Registers personal access token for Databricks to do operations on behalf of the user.
 * `bricks global-init-scripts` - The Global Init Scripts API enables Workspace administrators to configure global initialization scripts for their workspace.
 * `bricks grants` - In Unity Catalog, data is secure by default.
 * `bricks groups` - Groups simplify identity management, making it easier to assign access to Databricks Workspace, data, and other securable objects.
 * `bricks instance-pools` - Instance Pools API are used to create, edit, delete and list instance pools by using ready-to-use cloud instances which reduces a cluster start and auto-scaling times.
 * `bricks instance-profiles` - The Instance Profiles API allows admins to add, list, and remove instance profiles that users can launch clusters with.
 * `bricks ip-access-lists` - IP Access List enables admins to configure IP access lists.
 * `bricks jobs` - The Jobs API allows you to create, edit, and delete jobs.
 * `bricks libraries` - The Libraries API allows you to install and uninstall libraries and get the status of libraries on a cluster.
 * `bricks metastores` - A metastore is the top-level container of objects in Unity Catalog.
 * `bricks model-registry` - MLflow Model Registry commands.
 * `bricks permissions` - Permissions API are used to create read, write, edit, update and manage access for various users on different objects and endpoints.
 * `bricks pipelines` - The Delta Live Tables API allows you to create, edit, delete, start, and view details about pipelines.
 * `bricks policy-families` - View available policy families.
 * `bricks providers` - Databricks Providers REST API.
 * `bricks queries` - These endpoints are used for CRUD operations on query definitions.
 * `bricks query-history` - Access the history of queries through SQL warehouses.
 * `bricks recipient-activation` - Databricks Recipient Activation REST API.
 * `bricks recipients` - Databricks Recipients REST API.
 * `bricks repos` - The Repos API allows users to manage their git repos.
 * `bricks schemas` - A schema (also called a database) is the second layer of Unity Catalog’s three-level namespace.
 * `bricks secrets` - The Secrets API allows you to manage secrets, secret scopes, and access permissions.
 * `bricks service-principals` - Identities for use with jobs, automated tools, and systems such as scripts, apps, and CI/CD platforms.
 * `bricks serving-endpoints` - The Serving Endpoints API allows you to create, update, and delete model serving endpoints.
 * `bricks shares` - Databricks Shares REST API.
 * `bricks storage-credentials` - A storage credential represents an authentication and authorization mechanism for accessing data stored on your cloud tenant.
 * `bricks table-constraints` - Primary key and foreign key constraints encode relationships between fields in tables.
 * `bricks tables` - A table resides in the third layer of Unity Catalog’s three-level namespace.
 * `bricks token-management` - Enables administrators to get all tokens and delete tokens for other users.
 * `bricks tokens` - The Token API allows you to create, list, and revoke tokens that can be used to authenticate and access Databricks REST APIs.
 * `bricks users` - User identities recognized by Databricks and represented by email addresses.
 * `bricks volumes` - Volumes are a Unity Catalog (UC) capability for accessing, storing, governing, organizing and processing files.
 * `bricks warehouses` - A SQL warehouse is a compute resource that lets you run SQL commands on data objects within Databricks SQL.
 * `bricks workspace` - The Workspace API allows you to list, import, export, and delete notebooks and folders.
 * `bricks workspace-conf` - This API allows updating known workspace settings for advanced users.

## Account-level command groups

 * `bricks account billable-usage` - This API allows you to download billable usage logs for the specified account and date range.
 * `bricks account budgets` - These APIs manage budget configuration including notifications for exceeding a budget for a period.
 * `bricks account credentials` - These APIs manage credential configurations for this workspace.
 * `bricks account custom-app-integration` - These APIs enable administrators to manage custom oauth app integrations, which is required for adding/using Custom OAuth App Integration like Tableau Cloud for Databricks in AWS cloud.
 * `bricks account encryption-keys` - These APIs manage encryption key configurations for this workspace (optional).
 * `bricks account groups` - Groups simplify identity management, making it easier to assign access to Databricks Account, data, and other securable objects.
 * `bricks account ip-access-lists` - The Accounts IP Access List API enables account admins to configure IP access lists for access to the account console.
 * `bricks account log-delivery` - These APIs manage log delivery configurations for this account.
 * `bricks account metastore-assignments` - These APIs manage metastore assignments to a workspace.
 * `bricks account metastores` - These APIs manage Unity Catalog metastores for an account.
 * `bricks account networks` - These APIs manage network configurations for customer-managed VPCs (optional).
 * `bricks account o-auth-enrollment` - These APIs enable administrators to enroll OAuth for their accounts, which is required for adding/using any OAuth published/custom application integration.
 * `bricks account private-access` - These APIs manage private access settings for this account.
 * `bricks account published-app-integration` - These APIs enable administrators to manage published oauth app integrations, which is required for adding/using Published OAuth App Integration like Tableau Cloud for Databricks in AWS cloud.
 * `bricks account service-principals` - Identities for use with jobs, automated tools, and systems such as scripts, apps, and CI/CD platforms.
 * `bricks account storage` - These APIs manage storage configurations for this workspace.
 * `bricks account storage-credentials` - These APIs manage storage credentials for a particular metastore.
 * `bricks account users` - User identities recognized by Databricks and represented by email addresses.
 * `bricks account vpc-endpoints` - These APIs manage VPC endpoint configurations for this account.
 * `bricks account workspace-assignment` - The Workspace Permission Assignment API allows you to manage workspace permissions for principals in your account.
 * `bricks account workspaces` - These APIs manage workspaces for this account.
2023-04-26 13:06:16 +02:00
shreyas-goenka 43bc9a0d9d
Use cmdio logger to log bricks cmd execution errors (#348)
## Changes
Uses the cmdio logger to log the execution error

## Tests
Manually by making the root command return fake errors. Here is the
output:
```
shreyas.goenka@THW32HFW6T bricks % bricks bundle validate
Error: my foo error
```

```
shreyas.goenka@THW32HFW6T bricks % bricks bundle validate --progress-format=json
{
  "error": "my foo error"
}
```

---------

Co-authored-by: Pieter Noordhuis <pieter.noordhuis@databricks.com>
2023-04-24 12:11:52 +02:00
Pieter Noordhuis c26c7d388a
Update command descriptions (#353)
Cool tagline to be determined.
2023-04-21 13:41:25 +02:00
Pieter Noordhuis 81b69094aa
Remove stale command (#352) 2023-04-21 13:39:54 +02:00
Serge Smertin 9581187c9e
Update to Go SDK v0.8.0 (#351)
## Changes

- Update to Go SDK v0.8.0
- Fix all breaking changes

## Tests

- make test
2023-04-21 10:30:20 +02:00
shreyas-goenka 598ad62688
Log mutator messages using progress logger (#312)
This PR uses progress logger to log messages inside mutators
2023-04-18 16:55:06 +02:00
shreyas-goenka 1fc903943d
Log os.Args, bricks version, and exit status (#324)
## Changes
<!-- Summary of your changes that are easy to understand -->
1. Log os.Args and bricks version before every command execution
2. After a command execution, logs the error and exit code

## Tests
<!-- How is this tested? -->
Manually, 

case 1: Run `bricks version` successfully
```
shreyas.goenka@THW32HFW6T bricks % bricks version --log-level=info --log-file stderr
time=2023-04-12T00:15:04.011+02:00 level=INFO source=root.go:34 msg="process args: [bricks, version, --log-level=info, --log-file, stderr]"
time=2023-04-12T00:15:04.011+02:00 level=INFO source=root.go:35 msg="version: 0.0.0-dev+375eb1c50283"
0.0.0-dev+375eb1c50283
time=2023-04-12T00:15:04.011+02:00 level=INFO source=root.go:68 msg="exit code: 0"
```

case 2: Run `bricks bundle deploy` in a working dir where `bundle.yml`
does not exist
```
shreyas.goenka@THW32HFW6T bricks % bricks bundle deploy --log-level=info --log-file=stderr
time=2023-04-12T00:19:16.783+02:00 level=INFO source=root.go:34 msg="process args: [bricks, bundle, deploy, --log-level=info, --log-file=stderr]"
time=2023-04-12T00:19:16.784+02:00 level=INFO source=root.go:35 msg="version: 0.0.0-dev+375eb1c50283"
Error: unable to locate bundle root: bundle.yml not found
time=2023-04-12T00:19:16.784+02:00 level=ERROR source=root.go:64 msg="unable to locate bundle root: bundle.yml not found"
time=2023-04-12T00:19:16.784+02:00 level=ERROR source=root.go:65 msg="exit code: 1"
```
2023-04-12 22:12:36 +02:00
Pieter Noordhuis b388f4a0dc
Make all workspace paths string fields (#327)
## Changes

These are unlikely to ever be DBFS paths so we can remove this level of indirection to simplify.

**Note:** this is a breaking change. Downstream usage of these fields must be updated.

## Tests

Existing tests pass.
2023-04-12 16:54:36 +02:00
shreyas-goenka d52fc12644
Disable bricks fs and configure commands (#320)
## Changes
<!-- Summary of your changes that are easy to understand -->

## Tests
<!-- How is this tested? -->
2023-04-12 00:35:16 +02:00
shreyas-goenka 375eb1c502
Remove package project (#321)
## Changes
<!-- Summary of your changes that are easy to understand -->

This PR removes the project package and it's dependents in the bricks
repo

## Tests
<!-- How is this tested? -->
2023-04-11 16:59:27 +02:00
shreyas-goenka 4871f7bc8a
Add bundle destroy command (#300)
Adds bundle destroy capability to bricks
2023-04-06 12:54:58 +02:00
Pieter Noordhuis 04e77102c9
Add mutators to pull and push Terraform state (#288)
## Changes

Pull state before deploying and push state after deploying.

Note: the run command was missing mutators to initialize Terraform. This
is necessary if the cache directory is removed between running "deploy"
and "run" (which is valid now that we synchronize state).

## Tests

Manually.
2023-03-30 12:01:09 +02:00
shreyas-goenka 8fd3dccca9
Add progress logs for job runs (#276) 2023-03-29 14:58:09 +02:00
Pieter Noordhuis 1b47dd3af7
Trim log source field to basename of file (#273)
This makes logs more readable and avoids leaking paths.

Before:
```
time=2023-03-22T16:38:30.238+01:00 level=INFO source=/Users/pieter.noordhuis/dev/bricks/bundle/phases/phase.go:30 msg="Phase: initialize"
time=2023-03-22T16:38:31.303+01:00 level=INFO source=/Users/pieter.noordhuis/dev/bricks/bundle/phases/phase.go:30 msg="Phase: build"
time=2023-03-22T16:38:31.303+01:00 level=INFO source=/Users/pieter.noordhuis/dev/bricks/bundle/phases/phase.go:30 msg="Phase: deploy"
```

After:
```
time=2023-03-22T17:02:47.290+01:00 level=INFO source=phase.go:30 msg="Phase: initialize"
time=2023-03-22T17:02:48.171+01:00 level=INFO source=phase.go:30 msg="Phase: build"
time=2023-03-22T17:02:48.171+01:00 level=INFO source=phase.go:30 msg="Phase: deploy"
```
2023-03-23 08:56:39 +01:00
Pieter Noordhuis 123a5e15e9
Acquire lock prior to deploy (#270)
Add configuration:

```
bundle:
  lock:
    enabled: true
    force: false
```

The force field can be set by passing the `--force` argument to `bricks
bundle deploy`. Doing so means the deployment lock is acquired even if
it is currently held. This should only be used in exceptional cases
(e.g. a previous deployment has failed to release the lock).
2023-03-22 16:37:26 +01:00
Pieter Noordhuis 9100680162
Allow logger defaults to be configured through environment variables (#266)
These environment variables configure defaults for the logger related
flags:
* `BRICKS_LOG_FILE`
* `BRICKS_LOG_LEVEL`
* `BRICKS_LOG_FORMAT`
2023-03-21 17:05:04 +01:00
shreyas-goenka 047a189c1e
Add job run output logging (#260)
This PR adds output logging for job runs

Tested using unit tests and manually
2023-03-21 16:25:18 +01:00
Pieter Noordhuis ad666ff796
Use new logger throughout codebase (#256) 2023-03-17 15:17:31 +01:00
Pieter Noordhuis c9340d6317
Drain sync event channel before returning (#253)
Not waiting means the last few events may or may not be printed.
This is relevant in the mode where sync runs once and then terminates.
2023-03-16 17:48:17 +01:00
Pieter Noordhuis 32a29c6af4
Add structured logging infrastructure (#246)
New global flags:
* `--log-file FILE`: can be literal `stdout`, `stderr`, or a file name (default `stderr`)
* `--log-level LEVEL`: can be `error`, `warn`, `info`, `debug`, `trace`, or `disabled` (default `disabled`)
* `--log-format TYPE`: can be `text` or `json` (default `text`)

New functions in the `log` package take a `context.Context` and retrieve
the logger from said context.

Because we carry the logger in a context, adding
[attributes](https://pkg.go.dev/golang.org/x/exp/slog#hdr-Attrs_and_Values)
to the logger can be done as follows:

```go
ctx = log.NewContext(ctx, log.GetLogger(ctx).With("foo", "bar"))
```
2023-03-16 14:46:53 +01:00
shreyas-goenka 18a216bf97
Add openapi descriptions to bundle resources (#229)
This PR:
1. Adds autogeneration of descriptions for `resources` field
2. Autogenerates empty descriptions for any properties in DABs
3. Defines SOPs for how to refresh these descriptions
4. Adds command to generate this documentation
5. Adds Automatically copy any descriptions over to `environments`
property

Basically it provides a framework for adding descriptions to the
generated JSON schema

Tested manually and using unit tests
2023-03-15 03:18:51 +01:00
shreyas-goenka c4c8f944f3
Remove redundant terraform initialize mutator (#238)
Tested manually that bricks bundle run runs a pipeline.

phases.Initialize already has terraform.Initialize mutator
2023-03-09 15:05:02 +01:00
Pieter Noordhuis 46cfa747ac
Move and hide launch and test commands (#222)
Semantics in the context of a bundle are not yet clearly defined. Moving
and hiding these commands until then.
2023-03-09 10:26:56 +01:00
Pieter Noordhuis e872b587cc
Add optional JSON output for sync command (#230)
JSON output makes it easy to process synchronization progress
information in downstream tools (e.g. the vscode extension).
This changes introduces a `sync.Event` interface type for progress events as
well as an `sync.EventNotifier` that lets the sync code pass along
progress events to calling code.

Example output in text mode (default, this uses the existing logger calls):
```text
2023/03/03 14:07:17 [INFO] Remote file sync location: /Repos/pieter.noordhuis@databricks.com/...
2023/03/03 14:07:18 [INFO] Initial Sync Complete
2023/03/03 14:07:22 [INFO] Action: PUT: foo
2023/03/03 14:07:23 [INFO] Uploaded foo
2023/03/03 14:07:23 [INFO] Complete
2023/03/03 14:07:25 [INFO] Action: DELETE: foo
2023/03/03 14:07:25 [INFO] Deleted foo
2023/03/03 14:07:25 [INFO] Complete
```

Example output in JSON mode:
```json
{"timestamp":"2023-03-03T14:08:15.459439+01:00","seq":0,"type":"start"}
{"timestamp":"2023-03-03T14:08:15.459461+01:00","seq":0,"type":"complete"}
{"timestamp":"2023-03-03T14:08:18.459821+01:00","seq":1,"type":"start","put":["foo"]}
{"timestamp":"2023-03-03T14:08:18.459867+01:00","seq":1,"type":"progress","action":"put","path":"foo","progress":0}
{"timestamp":"2023-03-03T14:08:19.418696+01:00","seq":1,"type":"progress","action":"put","path":"foo","progress":1}
{"timestamp":"2023-03-03T14:08:19.421397+01:00","seq":1,"type":"complete","put":["foo"]}
{"timestamp":"2023-03-03T14:08:22.459238+01:00","seq":2,"type":"start","delete":["foo"]}
{"timestamp":"2023-03-03T14:08:22.459268+01:00","seq":2,"type":"progress","action":"delete","path":"foo","progress":0}
{"timestamp":"2023-03-03T14:08:22.686413+01:00","seq":2,"type":"progress","action":"delete","path":"foo","progress":1}
{"timestamp":"2023-03-03T14:08:22.688989+01:00","seq":2,"type":"complete","delete":["foo"]}
```

---------

Co-authored-by: shreyas-goenka <88374338+shreyas-goenka@users.noreply.github.com>
2023-03-08 10:27:19 +01:00
Pieter Noordhuis ae9d6883ee
Complete argument for the environment flag (#221)
Command completion can be configured through `bricks completion`.
2023-02-20 21:56:31 +01:00
Pieter Noordhuis dd95668474
Complete positional argument to bundle run (#220)
Command completion can be configured through `bricks completion`.
2023-02-20 21:55:06 +01:00
Pieter Noordhuis 7bf212e54a
Path completion for sync command (#208)
Follow up to #207.
2023-02-20 14:31:59 +01:00
Pieter Noordhuis 1715a987cf
Make sync command work in bundle context; reorder args (#207)
Invoke with `bricks sync SRC DST`.

In bundle context `SRC` and `DST` arguments are taken from bundle configuration.

This PR adds `bricks bundle sync` to disambiguate between the two.
Once the VS Code extension is bundle aware they can again be consolidated.
Consolidating them today would regress the VS Code experience if a
`bundle.yml` file is present in the file tree.
2023-02-20 11:33:30 +01:00
Pieter Noordhuis 3851b59bbd
Move code for including command name in user agent (#203) 2023-02-15 10:33:35 +01:00
Pieter Noordhuis 2e01473902
Let caller set BRICKS_UPSTREAM for user agent (#196)
Example when called from vscode (and everything is hooked up):

```
> * User-Agent: bricks/0.0.21-devel databricks-sdk-go/0.2.0 go/1.19.4 os/darwin upstream/databricks-vscode
```
2023-02-03 17:05:58 +01:00
Pieter Noordhuis 9ca7f8a888
Configure user agent in root command (#195)
This configures the user agent with the bricks version and the name of
the command being executed.

Example user agent value:
```
> * User-Agent: bricks/0.0.21-devel databricks-sdk-go/0.2.0 go/1.19.4 os/darwin cmd/sync auth/pat
```

This is a follow up for #194.
2023-02-03 16:47:33 +01:00
Pieter Noordhuis 1c27f081e0
Include build information and add version command (#194)
Includes relevant fields listed on
https://goreleaser.com/customization/templates/ into build artifacts.

The version command outputs the version by default:
```
$ bricks version
0.0.21-devel
```

Or all build information if `--json` is specified:
```
$ bricks version --json
{
  "ProjectName": "bricks",
  "Version": "0.0.21-devel",
  "Branch": "version-info",
  "Tag": "v0.0.20",
  "ShortCommit": "193b56b",
  "FullCommit": "193b56b0929128c0836d35e913c46fd66fa2a93c",
  "CommitTime": "2023-02-02T22:04:42+01:00",
  "Summary": "v0.0.20-5-g193b56b",
  "Major": 0,
  "Minor": 0,
  "Patch": 20,
  "Prerelease": "",
  "IsSnapshot": true,
  "BuildTime": "2023-02-02T22:07:36+01:00"
}
```
2023-02-03 15:38:53 +01:00
Pieter Noordhuis 241562e2b1
Move git package to libs/git (#189)
Fixes #185.
2023-01-31 19:19:16 +01:00
Pieter Noordhuis 6737af4b06
Move bundle loading functions to top level (#181)
We intend to let non-bundle commands use bundle configuration for their
operating context (workspace, auth, default cluster, etc).

As such, all commands must first try to load a bundle configuration.
If there is no bundle they can fall back on taking their operating
context from command line flags and the environment.

This is on top of #180.
2023-01-27 17:05:57 +01:00
Pieter Noordhuis 9a1d908f79
Add function to opportunistically load a bundle (#180)
It is not an error if a bundle cannot be found for this category.
This sets the stage for using bundle configuration in non-bundle
commands.
2023-01-27 16:57:39 +01:00
Fabian Jakobs b97e90acf1
bricks auth profiles tweaks (#178)
- add `validate` flag
- return empty list if config file doesn't exist
2023-01-24 15:54:28 +01:00
Pieter Noordhuis 03c863f49b
Update sync defaults (#177)
By default the command runs an incremental, one-time sync, similar to the
behavior of rsync. The `--persist-snapshot` flag has been removed and the
command now always saves a synchronization snapshot.

* Add `--full` flag to force full synchronization
* Add `--watch` flag to run continuously and watch the local file system for changes

This builds on #176.
2023-01-24 15:06:59 +01:00
Pieter Noordhuis 077304ffa1
Move path checking logic for sync command to libs/sync (#176)
This change also adds testcases for checking if the specified path is
nested under the valid base paths and fixes an edge case where the user
could synchronize into their home directory directly.

Co-authored-by: shreyas-goenka <88374338+shreyas-goenka@users.noreply.github.com>
2023-01-24 13:58:10 +01:00
Pieter Noordhuis 015a2bf9bb
Remove dependency on project package in libs/sync (#174)
The code depended on the project package for:
* git.FileSet in the watchdog
* project.CacheDir to determine snapshot path

These dependencies are now denormalized in the SyncOptions struct.

Follow up for #173.
2023-01-24 08:30:10 +01:00
shreyas-goenka 83fb89ad3b
Add command for generating JSON schema for DABs bundle config (#171)
In the future can add a path flag to generate subschemas. Might be
useful depending on how config splits are supported
2023-01-23 15:00:11 +01:00
Pieter Noordhuis fc46d21f8b
Move sync logic from cmd/sync to libs/sync (#173)
Mechanical change. Ported global variables the logic relied on to a new
`sync.Sync` struct.
2023-01-23 13:52:39 +01:00
Pieter Noordhuis 3b53b23b5b
Allow sync to workspace path (#170)
With this change:
* Paths under `/Workspace/<me>` and `/Repos/<me>` are allowed
* The sync destination is checked to be either a directory or a repository
* If it is under `/Repos` and doesn't exist, the command returns an error
2023-01-19 15:57:41 +01:00
shreyas-goenka 0d9ecb5643
Refactor and cover edge cases in sync integration tests (#160)
This PR:
1. Refactors the sync integration tests to make them more readable
2. Adds additional tests for edge cases we encountered during vscode
runs
3. Intensional side effect: sync integration tests are also green on
windows (see
https://github.com/databricks/eng-dev-ecosystem/actions/runs/3817365642/jobs/6493576727)

Change in coverage

- We now test for python notebook <-> python file interconversion and
python notebook deletion being synced to workspace
- Tests are split up and are more focused on testing specific edge cases
2023-01-10 13:16:30 +01:00
Serge Smertin b87b4b0f40
Added `bricks auth login` and `bricks auth token` (#158)
# Auth challenge (happy path)

Simplified description of [PKCE](https://oauth.net/2/pkce/)
implementation:

```mermaid
sequenceDiagram
    autonumber
    actor User
    
    User ->> CLI: type `bricks auth login HOST`
    CLI ->>+ HOST: request OIDC endpoints
    HOST ->>- CLI: auth & token endpoints
    CLI ->> CLI: start embedded server to consume redirects (lock)
    CLI -->>+ Auth Endpoint: open browser with RND1 + SHA256(RND2)

    User ->>+ Auth Endpoint: Go through SSO
    Auth Endpoint ->>- CLI: AUTH CODE + 'RND1 (redirect)

    CLI ->>+ Token Endpoint: Exchange: AUTH CODE + RND2
    Token Endpoint ->>- CLI: Access Token (JWT) + refresh + expiry
    CLI ->> Token cache: Save Access Token (JWT) + refresh + expiry
    CLI ->> User: success
```

# Token refresh (happy path)

```mermaid
sequenceDiagram
    autonumber
    actor User
    
    User ->> CLI: type `bricks token HOST`
    
    CLI ->> CLI: acquire lock (same local addr as redirect server)
    CLI ->>+ Token cache: read token

    critical token not expired
    Token cache ->>- User: JWT (without refresh)

    option token is expired
    CLI ->>+ HOST: request OIDC endpoints
    HOST ->>- CLI: auth & token endpoints
    CLI ->>+ Token Endpoint: refresh token
    Token Endpoint ->>- CLI: JWT (refreshed)
    CLI ->> Token cache: save JWT (refreshed)
    CLI ->> User: JWT (refreshed)
    
    option no auth for host
    CLI -X User: no auth configured
    end
```
2023-01-06 16:15:57 +01:00
Pieter Noordhuis 8f4461904b
Define flags for running jobs and pipelines (#146) 2022-12-23 15:17:16 +01:00
Pieter Noordhuis 49aa858b89
Run command must always take a single argument (#156) 2022-12-22 16:19:38 +01:00
shreyas-goenka 198eefcf39
Fix folder syncing on windows (#149)
Tested by running the unit and integration tests locally

Tested manually on windows

Screenshot from windows sync logs indicating that the correct slashed
for paths were used:
<img width="623" alt="Screenshot 2022-12-21 at 9 09 13 PM"
src="https://user-images.githubusercontent.com/88374338/208943937-146670b2-1afd-4e0b-8f4e-6091c8c7e17a.png">

@pietern with this the state machine for syncing becomes slightly more
complicated, indicating a stronger need for a tree based approach herre

Co-authored-by: Pieter Noordhuis <pieter.noordhuis@databricks.com>
2022-12-22 11:35:32 +01:00
Pieter Noordhuis a354fa1f77
Only display usage string on flag errors (#147) 2022-12-21 11:38:30 +01:00
Pieter Noordhuis f70a5814f6
Run initialize phase from validate command (#145)
This lets the user know if their workspace connection is setup
correctly.
2022-12-16 15:00:48 +01:00
Pieter Noordhuis 24a3b90713
Add "default" flag to environment block (#142)
If the environment is not set through command line argument or
environment variable, the bundle loads either 1) the only environment,
2) the only environment with the default flag set.
2022-12-15 21:28:14 +01:00
Pieter Noordhuis b111416fe5
Add `bricks bundle run` command (#134) 2022-12-15 15:12:47 +01:00
shreyas-goenka b42768801d
[DECO-396] Post mortem followups on sync deletes repo (#119)
This PR:
- Implements safeguards for not accidentally/maliciously deleting repos
by sanitizing relative paths
- Adds versioning for snapshot schemas to allow invalidation if needed
- Adds logic to delete preexisting remote artifacts that might not have
been cleaned up properly if they conflict with an upload
- A bunch of tests for the changes here

Co-authored-by: Pieter Noordhuis <pieter.noordhuis@databricks.com>
2022-12-12 14:31:06 +01:00
Pieter Noordhuis c255bd686a
Define deploy command as sequence of build phases (#129) 2022-12-12 12:49:25 +01:00
shreyas-goenka d9d295f2a9
Implement Terraform state synchronization and deploy (#98)
https://user-images.githubusercontent.com/88374338/203669797-abebf99e-8fa6-4d6e-b57a-abd172d8020d.mov
2022-12-06 00:40:45 +01:00
shreyas-goenka c6b3b35e98
[DECO-396] Send delete file requests with recursive set to false (#106)
Safeguard so bugs do not delete large amount of remote files
2022-11-30 13:56:52 +01:00
shreyas-goenka 2ebfa5f369
Run unit tests on windows and macos (#103)
Unit tests are now run in all three big OS. 

Some of the changes are to make the tests green for windows while we are
skipping some of the other tests on windows/macOS to make the tests
pass. This is a temporary measure and we will incrementally migrate
these tests over so there is parity in unit testing along all three
environments!
2022-11-28 11:34:25 +01:00
Pieter Noordhuis b88b35a510
Move mutator interface to top level bundle package (#105)
While working on artifact upload and workspace interrogation I realized
this mutator interface needs to:
1. Operate at the whole bundle level so it can apply to both
configuration and internal state
2. Include a `context.Context` parameter for a) long running operations
and b) progress reporting

Previous interface:
```
Apply(*config.Root) ([]Mutator, error)
```

New interface:
```
Apply(context.Context, *Bundle) ([]Mutator, error)
```
2022-11-28 10:59:43 +01:00
shreyas-goenka 761de12a0d
Fix incorrect remote path arg to APIs in windows on bricks sync (#101)
Integration tests testing sync on ubuntu are green:
<img width="794" alt="Screenshot 2022-11-25 at 2 05 11 PM"
src="https://user-images.githubusercontent.com/88374338/203991936-f0478c35-5d97-429b-b373-f050d9a90cff.png">

@fjakobs tested this on windows manually
2022-11-25 14:13:15 +01:00
Pieter Noordhuis 8e786d76a9
Update databricks-sdk-go to latest (#102) 2022-11-24 21:41:57 +01:00
Pieter Noordhuis 07f07694a4
Function to return workspace client on bundle.Bundle (#100)
Complementary command to check the identity in the context of a bundle
environment:

For example:
```
bricks bundle debug whoami -e development
```
2022-11-23 15:20:03 +01:00
shreyas-goenka b25bd318e3
Deal with notebook remote filenames stripping .py (#90)
This PR introduces tracking of remote names and local names of files in snapshots to disambiguate between files which might have the same remote name and handle clean deleting of files whose remote name changes due (eg. python notebook getting converted to a python notebook)
2022-11-21 23:42:09 +01:00
Pieter Noordhuis 3b351d3b00
Add command that writes the materialized bundle configuration to stdout (#95)
Used to inspect the bundle configuration after loading and merging all
files.

Once we add variable interpolation this command could show the result
after interpolation as well.

Each of the mutations to this configuration is observable, so we could
add a mode that writes each of the intermediate versions to disk for
even more fine grained introspection.
2022-11-21 15:39:53 +01:00
Shreyas Goenka 0c24e6f82e
Revert "WIP initial version of the workspace file lock done"
This reverts commit 02eec1f990.
2022-11-16 23:50:17 +01:00
Shreyas Goenka 02eec1f990
WIP initial version of the workspace file lock done 2022-11-16 17:30:46 +01:00
Pieter Noordhuis af5659ae8e
Update databricks-sdk-go to latest (#89) 2022-11-09 15:01:47 +01:00
Shreyas Goenka efe1f28b36
Revert "Added creation of .gitignore for bricks project with cache dir path"
This reverts commit 1505c63426.
2022-11-03 21:09:29 +01:00
Shreyas Goenka 1505c63426
Added creation of .gitignore for bricks project with cache dir path 2022-11-03 21:06:21 +01:00
shreyas-goenka 0601e114fd
[DECO-200] Ignore RESOURCE_DOES_NOT_EXIST errors on file deletion during sync (#85)
tested manually
2022-10-27 17:32:10 +02:00
shreyas-goenka 1329bc05d4
Log initial sync complete (#86)
Tested manually
<img width="930" alt="Screenshot 2022-10-27 at 2 21 52 PM"
src="https://user-images.githubusercontent.com/88374338/198282897-ad0c3d62-be92-4ec4-a0e1-4df2a7eab2b9.png">
2022-10-27 15:41:18 +02:00
shreyas-goenka 18dae73505
[DECO-79][DECO-165] Incremental sync with support for multiple profiles (#82)
This PR does multiple things, which are:

1. Creates .databricks dir according to outcomes concluded in "bricks
configuration principles"
2. Puts the sync snapshots into a file whose names is tagged with
md5(concat(host, remote-path))
3. Saves both host and username in the bricks snapshot for debuggability

Tested manually:


https://user-images.githubusercontent.com/88374338/195672267-9dd90230-570f-49b7-847f-05a5a6fd8986.mov
2022-10-19 16:22:55 +02:00
Pieter Noordhuis 38a9dabcbe
Add command to make API calls (#80)
Not settled whether this should live as a top level command or hidden
under some debug scope. Either way, the ability to make arbitrary API
calls and leverage unified auth is a super useful tool.
2022-10-10 10:27:45 +02:00
shreyas-goenka 0b754e6de8
[DECO-94] Execute uploads in parallel instead of sequentailly (#81)
Tested manually

Upload seems fast enough, delete API calls though have a much longer
turn around times

Additional optimizations that can be done if/when the need arises:
1. First time upload can be done using zip batching of the files


https://user-images.githubusercontent.com/88374338/192783332-9b2b19bc-d6c4-4a66-8dbc-e78287e6af1a.mov
2022-10-05 00:12:57 +02:00
Pieter Noordhuis a1b6fdb2e8
Update SDK (#79) 2022-09-27 09:58:55 -07:00
Pieter Noordhuis 6258a1637d
Add environments to project configuration (#68) 2022-09-22 13:40:11 +02:00
shreyas-goenka 731679cb4b
Add `persist-snapshot` to bricks sync (#66)
Tested manually

We are adding this flag because the default bricks sync is not robust
against changing the profile and other project config changes. This will
be used in the initial version of the vscode extention
2022-09-19 16:47:55 +02:00
Pieter Noordhuis 7cad8bda81
Respect project root in sync command (#63) 2022-09-16 15:18:46 +02:00
Pieter Noordhuis a7701cc8f3
Store project object in context.Context instead of global (#61)
* Load project root from `BRICKS_ROOT` environment variable
* Rename project.Project -> project.Config
* Rename project.inner -> project.project
* Upgrade cobra to 1.5.0 for cmd.SetContext
2022-09-16 11:06:58 +02:00
shreyas-goenka 836ab58473
Fix flaky test `TestDiff` (#59) 2022-09-15 15:40:47 +02:00
shreyas-goenka f9b66b3536
Make `bricks sync` feature work (#48)
Tested manually and partially by unit tests
2022-09-14 17:50:29 +02:00
shreyas-goenka 21bc774491
Replace scim Me terraform call with go sdk (#46)
This PR:
1. Replaces scim.Me call to use the go SDK instead of the terraform
client
2. Removes terraform client from bricks project

Tested manually that the scim.Me call works now and returns the correct
user
go build works
2022-09-08 15:50:00 +02:00
Pieter Noordhuis 2e12a2aa01
Make tests pass (#40)
By:
* Add .gitkeep to retain test fixture directories under
./python/testdata
* Move GitHub related functionality to ./experimental (it is not in use)
* Comment out test in ./cmd/sync
* Fix test in ./git
2022-09-07 20:08:42 +02:00
Pieter Noordhuis 80a4c47d62
Fix lint error regarding struct with unexported fields (#38)
Unexported fields are skipped during marshalling, so this would be a
nop.

The code in `./cmd/sync/github*.go` is currently unused.

Confirmed that staticcheck passes when run with:

```
staticcheck -checks SA9005 ./...
```
2022-09-07 15:15:07 +02:00
Pieter Noordhuis c00db56d84
Fix lint errors for using deprecated functionality (#35)
Functionality from `io/ioutil` has moved to the `io` and `os` packages
in go1.16 ([reference](https://pkg.go.dev/io/ioutil)).

Confirmed that staticcheck passes when run with:

```
staticcheck -checks SA1019 ./...
```
2022-09-07 14:30:10 +02:00
Pieter Noordhuis 67b2e4206f
Use t.Setenv instead of custom implementation (#34)
Follow up for #18. This function was introduced in go1.17. As of #19 we
require go1.18, so we can use it.
2022-09-07 13:24:11 +02:00
shreyas-goenka 96efd0e2e4
Replace terraform dependency with go sdk (#19)
Issue: https://github.com/databricks/bricks/issues/17

`./bricks fs ls ...` command works
`./bricks launch ...` command works

Did not test other changes as the readme claims other commands don't
work anyways :) cc: @nfx

TODO left for this PR:
2. Replace terraform scim.Me once its there in go SDK
(https://github.com/databricks/databricks-sdk-go/issues/56)
2022-09-07 11:55:59 +02:00
Kartik Gupta f070c24399
Write default header only if `DEFAULT` section exists (#31) 2022-09-06 18:04:05 +02:00
Kartik Gupta 30a7de621a
Add more flags to `configure` command (#29) 2022-09-06 16:37:58 +02:00
Kartik Gupta 457f3ad3c2
Add `bricks configure` command to bricks CLI (#18)
* bricks configure

* remove t.setenv

* Read token and host from stdin

* Update .vscode/testing.code-snippets

Co-authored-by: Serge Smertin <259697+nfx@users.noreply.github.com>

Co-authored-by: Serge Smertin <259697+nfx@users.noreply.github.com>
2022-09-05 20:25:54 +02:00
Shreyas Goenka 9951068c62
Revert "Using go sdk for ./bricks fs ls DIR_NAME command"
This reverts commit 6217d20e57.
2022-09-01 13:42:34 +02:00
Shreyas Goenka 6217d20e57
Using go sdk for ./bricks fs ls DIR_NAME command 2022-08-31 19:31:58 +02:00
Serge Smertin 32ae59c1bc Experimental sync command 2022-07-07 20:56:59 +02:00
Serge Smertin ae2dc104f9 add some comments to commands package 2022-05-20 20:43:29 +02:00
Serge Smertin 1b38687e2d more TODO's 2022-05-16 12:50:50 +02:00
Serge Smertin 154fb8a967 added basic init command for `databricks.yml` 2022-05-14 19:55:55 +02:00
Serge Smertin 4e8955085e moved commands to own packages 2022-05-14 19:54:35 +02:00
Serge Smertin 2dab552829 Added `launch` command and release pipeline 2022-05-13 17:43:54 +02:00
Serge Smertin 02e62a7176 added `fs ls` command prototype 2022-05-13 16:21:47 +02:00
Serge Smertin 15fd93a012 Initial commit 2022-05-13 15:30:22 +02:00