Commit Graph

352 Commits

Author SHA1 Message Date
Fabian Jakobs da4b58a897
Fix link to workspace after AWS OAuth login (#234)
`Host` is already normalized and always has the `https://` prefix.
2023-03-08 11:56:46 +01:00
Pieter Noordhuis e872b587cc
Add optional JSON output for sync command (#230)
JSON output makes it easy to process synchronization progress
information in downstream tools (e.g. the vscode extension).
This changes introduces a `sync.Event` interface type for progress events as
well as an `sync.EventNotifier` that lets the sync code pass along
progress events to calling code.

Example output in text mode (default, this uses the existing logger calls):
```text
2023/03/03 14:07:17 [INFO] Remote file sync location: /Repos/pieter.noordhuis@databricks.com/...
2023/03/03 14:07:18 [INFO] Initial Sync Complete
2023/03/03 14:07:22 [INFO] Action: PUT: foo
2023/03/03 14:07:23 [INFO] Uploaded foo
2023/03/03 14:07:23 [INFO] Complete
2023/03/03 14:07:25 [INFO] Action: DELETE: foo
2023/03/03 14:07:25 [INFO] Deleted foo
2023/03/03 14:07:25 [INFO] Complete
```

Example output in JSON mode:
```json
{"timestamp":"2023-03-03T14:08:15.459439+01:00","seq":0,"type":"start"}
{"timestamp":"2023-03-03T14:08:15.459461+01:00","seq":0,"type":"complete"}
{"timestamp":"2023-03-03T14:08:18.459821+01:00","seq":1,"type":"start","put":["foo"]}
{"timestamp":"2023-03-03T14:08:18.459867+01:00","seq":1,"type":"progress","action":"put","path":"foo","progress":0}
{"timestamp":"2023-03-03T14:08:19.418696+01:00","seq":1,"type":"progress","action":"put","path":"foo","progress":1}
{"timestamp":"2023-03-03T14:08:19.421397+01:00","seq":1,"type":"complete","put":["foo"]}
{"timestamp":"2023-03-03T14:08:22.459238+01:00","seq":2,"type":"start","delete":["foo"]}
{"timestamp":"2023-03-03T14:08:22.459268+01:00","seq":2,"type":"progress","action":"delete","path":"foo","progress":0}
{"timestamp":"2023-03-03T14:08:22.686413+01:00","seq":2,"type":"progress","action":"delete","path":"foo","progress":1}
{"timestamp":"2023-03-03T14:08:22.688989+01:00","seq":2,"type":"complete","delete":["foo"]}
```

---------

Co-authored-by: shreyas-goenka <88374338+shreyas-goenka@users.noreply.github.com>
2023-03-08 10:27:19 +01:00
shreyas-goenka 5166055efb
[DECO-553] Escape file path strings in URL (#228)
Tested manually

Before:
```
shreyas.goenka@THW32HFW6T test-dbx % bricks sync --full . /Repos/shreyas.goenka@databricks.com/test-dbx
2023/02/27 19:51:17 [INFO] Remote file sync location: /Repos/shreyas.goenka@databricks.com/test-dbx
2023/02/27 19:51:17 [INFO] Action: PUT: #foo.py, .gitignore
2023/02/27 19:51:19 [INFO] Uploaded .gitignore
Error: Creating file failed. An item with path /Repos/shreyas.goenka@databricks.com/test-dbx already exists
```

After:
```
shreyas.goenka@THW32HFW6T test-dbx % bricks sync --full . /Repos/shreyas.goenka@databricks.com/test-dbx
2023/02/27 19:51:46 [INFO] Remote file sync location: /Repos/shreyas.goenka@databricks.com/test-dbx
2023/02/27 19:51:46 [INFO] Action: PUT: #foo.py, .gitignore
2023/02/27 19:51:47 [INFO] Uploaded .gitignore
2023/02/27 19:51:47 [INFO] Uploaded #foo.py
```
2023-02-28 03:17:13 +01:00
shreyas-goenka 2615d66945
[DECO-531] Increase timeout for file import api calls (#223)
This PR increases the client side timeout for upload API calls to 10
minutes to give sync enough time to import larger files
2023-02-22 16:01:58 +01:00
Pieter Noordhuis 9d3a0da073
Detect Jupyter notebook files (#219)
Files with extension `.ipynb` are imported are Jupyter notebooks.

This code detects 1) if the file is a valid Jupyter notebook and 2) the
Databricks specific language it contains.
2023-02-21 13:49:01 +01:00
shreyas-goenka f93b541b63
Show detailed error logs for jobs (#209)
PR for how to render errors on console for jobs. 
Here is the bundle used for the logs below:
```
bundle:
  name: deco-438

workspace:
  host: https://adb-309687753508875.15.azuredatabricks.net

resources:
  jobs:
    foo:
      name: "[${bundle.name}][${bundle.environment}] a test notebook"

      tasks:
        - task_key: alpha
          existing_cluster_id: 1109-115254-ox7poobk
          notebook_task:
            notebook_path: "/Users/shreyas.goenka@databricks.com/[deco-438] invalid notebook"
        - task_key: beta
          existing_cluster_id: 1109-115254-ox7poobk
          notebook_task:
            notebook_path: "/does-not-exist"
        - task_key: gamma
          existing_cluster_id: 1109-115254-ox7poobk
          notebook_task:
            notebook_path: "/Users/shreyas.goenka@databricks.com/[deco-438] valid notebook"
```

And this is a screenshot of the logs from the console:
<img width="1057" alt="Screenshot 2023-02-17 at 7 12 29 PM"
src="https://user-images.githubusercontent.com/88374338/219744768-ab7f1e79-db8f-466a-ad6d-f2b6f85ed17c.png">

Here are the logs when only tasks gamma is executed (successfully):
<img width="1059" alt="Screenshot 2023-02-17 at 7 13 04 PM"
src="https://user-images.githubusercontent.com/88374338/219744992-011d8b91-ec1d-44f0-a849-83c81816dd9f.png">


TODO: Investigate more possible job errors, and make sure state for them
is handled in a robust way here
2023-02-20 23:40:14 +01:00
Pieter Noordhuis ae9d6883ee
Complete argument for the environment flag (#221)
Command completion can be configured through `bricks completion`.
2023-02-20 21:56:31 +01:00
Pieter Noordhuis dd95668474
Complete positional argument to bundle run (#220)
Command completion can be configured through `bricks completion`.
2023-02-20 21:55:06 +01:00
Pieter Noordhuis 9912ee1f92
Materialize glob expansion in configuration struct (#217)
This is needed to figure out which files should adhere to the schema.
2023-02-20 21:01:28 +01:00
Pieter Noordhuis 7398a6d1e4
Add sample ipynb files (#218)
Co-authored-by: pietern <pietern>
2023-02-20 20:03:20 +01:00
Pieter Noordhuis a0ed02281d
Execute file synchronization on deploy (#211)
1. Perform file synchronization on deploy
2. Update notebook file path translation logic to point to the
synchronization target rather than treating the notebook as an artifact
and uploading it separately.
2023-02-20 19:42:55 +01:00
Pieter Noordhuis 414ea4f891
Bump databricks-sdk-go to 0.3.2 (#215) 2023-02-20 16:00:20 +01:00
Pieter Noordhuis 5b56b3e815
Include commit hash in snapshot version (#193)
After this change the version for snapshot builds looks like
`0.0.21-dev+65020f3`.

This is valid semver per the regexp on https://semver.org/.
2023-02-20 15:46:57 +01:00
Pieter Noordhuis ca04a6a1dd
Fix sync test (miss in #207) (#216) 2023-02-20 15:41:37 +01:00
Pieter Noordhuis 584c8d1b0b
Allow synchronization to a directory inside a repo (#213)
Before this commit this would error saying that the repo doesn't exist yet.

With this commit it creates the directory, but only after checking that
the repo exists.
2023-02-20 14:34:48 +01:00
Pieter Noordhuis 7bf212e54a
Path completion for sync command (#208)
Follow up to #207.
2023-02-20 14:31:59 +01:00
Pieter Noordhuis 6c93c96bd1
Update deps for internal-only tree (#214)
Fixes dependabot warnings.
2023-02-20 14:30:42 +01:00
Pieter Noordhuis 1715a987cf
Make sync command work in bundle context; reorder args (#207)
Invoke with `bricks sync SRC DST`.

In bundle context `SRC` and `DST` arguments are taken from bundle configuration.

This PR adds `bricks bundle sync` to disambiguate between the two.
Once the VS Code extension is bundle aware they can again be consolidated.
Consolidating them today would regress the VS Code experience if a
`bundle.yml` file is present in the file tree.
2023-02-20 11:33:30 +01:00
dependabot[bot] ded7d8c404
Bump golang.org/x/net from 0.1.0 to 0.7.0 (#210) 2023-02-20 11:23:25 +01:00
dependabot[bot] 1810a8ba61
Bump github.com/hashicorp/terraform-json from 0.14.0 to 0.15.0 (#202) 2023-02-20 10:57:33 +01:00
dependabot[bot] a94d023776
Bump golang.org/x/text from 0.6.0 to 0.7.0 (#201) 2023-02-20 10:56:53 +01:00
dependabot[bot] 04f16b20b1
Bump golang.org/x/mod from 0.7.0 to 0.8.0 (#200) 2023-02-20 10:56:31 +01:00
dependabot[bot] aa4ba29879
Bump github.com/hashicorp/hc-install from 0.4.0 to 0.5.0 (#198) 2023-02-20 10:21:47 +01:00
shreyas-goenka 0ab2aa1bfa
Make file, artifact and state path optional (#204)
This PR makes bundle name required, and a few fields with defined
defaults optional, to generate a better json schema
2023-02-17 02:49:39 +01:00
Pieter Noordhuis 58950ce507
Move notebook detection logic to package (#206) 2023-02-15 17:14:59 +01:00
Fabian Jakobs 8c1b620b17
Don't sync symlink folders (#205)
Fixes https://github.com/databricks/databricks-vscode/issues/452
2023-02-15 17:02:54 +01:00
Pieter Noordhuis 3851b59bbd
Move code for including command name in user agent (#203) 2023-02-15 10:33:35 +01:00
Pieter Noordhuis 2e01473902
Let caller set BRICKS_UPSTREAM for user agent (#196)
Example when called from vscode (and everything is hooked up):

```
> * User-Agent: bricks/0.0.21-devel databricks-sdk-go/0.2.0 go/1.19.4 os/darwin upstream/databricks-vscode
```
2023-02-03 17:05:58 +01:00
Pieter Noordhuis 9ca7f8a888
Configure user agent in root command (#195)
This configures the user agent with the bricks version and the name of
the command being executed.

Example user agent value:
```
> * User-Agent: bricks/0.0.21-devel databricks-sdk-go/0.2.0 go/1.19.4 os/darwin cmd/sync auth/pat
```

This is a follow up for #194.
2023-02-03 16:47:33 +01:00
Pieter Noordhuis 1c27f081e0
Include build information and add version command (#194)
Includes relevant fields listed on
https://goreleaser.com/customization/templates/ into build artifacts.

The version command outputs the version by default:
```
$ bricks version
0.0.21-devel
```

Or all build information if `--json` is specified:
```
$ bricks version --json
{
  "ProjectName": "bricks",
  "Version": "0.0.21-devel",
  "Branch": "version-info",
  "Tag": "v0.0.20",
  "ShortCommit": "193b56b",
  "FullCommit": "193b56b0929128c0836d35e913c46fd66fa2a93c",
  "CommitTime": "2023-02-02T22:04:42+01:00",
  "Summary": "v0.0.20-5-g193b56b",
  "Major": 0,
  "Minor": 0,
  "Patch": 20,
  "Prerelease": "",
  "IsSnapshot": true,
  "BuildTime": "2023-02-02T22:07:36+01:00"
}
```
2023-02-03 15:38:53 +01:00
Pieter Noordhuis cd789366b9
Bump Go version to 1.19 (#197) 2023-02-03 15:13:29 +01:00
Pieter Noordhuis 3796b0a6b0
Revert snapshot name template to previous value (#192)
This was changed in #159 but not necessary to have a stable binary name.
2023-02-02 21:50:51 +01:00
Pieter Noordhuis abb1de99ba
Locate and use global excludes file (#191)
This implements rudimentary gitconfig loading as specified at
https://git-scm.com/docs/git-config.
2023-02-02 12:25:53 +01:00
Pieter Noordhuis 241562e2b1
Move git package to libs/git (#189)
Fixes #185.
2023-01-31 19:19:16 +01:00
Pieter Noordhuis a7bf7ba6c5
Reload .gitignore files if they have changed (#190)
This commit changes the code in repository.go to lazily load gitignore
files as opposed to the previous eager approach. This means that the
signature of the `Ignore` function family has changed to return `(bool,
error)`.

This lazy approach fits better when other code is responsible for
recursively walking the file tree, because we never know up front which
gitignore files need to be loaded to compute the ignores. It also means
we no longer have to "prime" the `Repository` instance with a particular
directory we're interested in and rather let calls to `Ignore` load
whatever is needed.

The fileset wrapper under `git/` internally taints all gitignore objects
to force a call to [os.Stat] followed by a reload if they have changed,
before calling into the [fileset.FileSet] functions for recursively
listing files.
2023-01-31 18:34:36 +01:00
Pieter Noordhuis 6737af4b06
Move bundle loading functions to top level (#181)
We intend to let non-bundle commands use bundle configuration for their
operating context (workspace, auth, default cluster, etc).

As such, all commands must first try to load a bundle configuration.
If there is no bundle they can fall back on taking their operating
context from command line flags and the environment.

This is on top of #180.
2023-01-27 17:05:57 +01:00
Pieter Noordhuis 9a1d908f79
Add function to opportunistically load a bundle (#180)
It is not an error if a bundle cannot be found for this category.
This sets the stage for using bundle configuration in non-bundle
commands.
2023-01-27 16:57:39 +01:00
Pieter Noordhuis 035fb6514d
Expand comments in git.View (#188) 2023-01-27 16:17:44 +01:00
Pieter Noordhuis eb76e5d3e8
Move git.FileSet to libs/fileset and make it aware of gitignores (#184)
This moves `git.FileSet` to `libs/fileset` and decouples it from the Git package.

It is made aware of gitignore rules in parent directories up to the
repository root as well as gitignore files in underlying directories
through the `fileset.Ignorer` interface.

The recursive directory walker is reimplemented with [filepath.WalkDir].

Follow up to #182.
2023-01-27 16:04:58 +01:00
Pieter Noordhuis 2eb10800a5
Take into account gitignore rules defined in parent directories (#182)
This change introduces `git.View`.

View represents a view on a directory tree that takes into account all
applicable .gitignore files. The directory tree does NOT need to be the
repository root.

For example: with a repository root at "myrepo", a view can be anchored
at "myrepo/someproject" and still respect the ignore rules defined at
"myrepo/.gitignore".

We use this functionality to synchronize files from a path nested in a
repository while respecting the repository's ignore rules.

Co-authored-by: Serge Smertin <259697+nfx@users.noreply.github.com>
2023-01-27 14:54:28 +01:00
Pieter Noordhuis df7b341afe
Remove unused function git.RepoExists (#183)
This function is no longer in use.

The SDK has native functionality for this through `GetByName`.

Closes #57.
2023-01-26 21:11:15 +01:00
Pieter Noordhuis 35c3d9fa4e
Add workspace paths (#179)
The workspace root path is a base path for bundle storage. If not
specified, it defaults to `~/.bundle/name/environment`. This default, or
other paths starting with `~` are expanded to the current user's home
directory. The configuration also includes fields for the files path,
artifacts path, and state path. By default, these are nested under the
root path, but can be overridden if needed.
2023-01-26 19:55:38 +01:00
Fabian Jakobs b97e90acf1
bricks auth profiles tweaks (#178)
- add `validate` flag
- return empty list if config file doesn't exist
2023-01-24 15:54:28 +01:00
Pieter Noordhuis 03c863f49b
Update sync defaults (#177)
By default the command runs an incremental, one-time sync, similar to the
behavior of rsync. The `--persist-snapshot` flag has been removed and the
command now always saves a synchronization snapshot.

* Add `--full` flag to force full synchronization
* Add `--watch` flag to run continuously and watch the local file system for changes

This builds on #176.
2023-01-24 15:06:59 +01:00
Pieter Noordhuis 077304ffa1
Move path checking logic for sync command to libs/sync (#176)
This change also adds testcases for checking if the specified path is
nested under the valid base paths and fixes an edge case where the user
could synchronize into their home directory directly.

Co-authored-by: shreyas-goenka <88374338+shreyas-goenka@users.noreply.github.com>
2023-01-24 13:58:10 +01:00
Pieter Noordhuis c777a703cf
Move diff struct to its own file (#175) 2023-01-24 11:06:14 +01:00
Pieter Noordhuis 015a2bf9bb
Remove dependency on project package in libs/sync (#174)
The code depended on the project package for:
* git.FileSet in the watchdog
* project.CacheDir to determine snapshot path

These dependencies are now denormalized in the SyncOptions struct.

Follow up for #173.
2023-01-24 08:30:10 +01:00
shreyas-goenka 83fb89ad3b
Add command for generating JSON schema for DABs bundle config (#171)
In the future can add a path flag to generate subschemas. Might be
useful depending on how config splits are supported
2023-01-23 15:00:11 +01:00
Pieter Noordhuis fc46d21f8b
Move sync logic from cmd/sync to libs/sync (#173)
Mechanical change. Ported global variables the logic relied on to a new
`sync.Sync` struct.
2023-01-23 13:52:39 +01:00
Pieter Noordhuis f122e29279
Let vscode trim whitespace on save (#172) 2023-01-23 09:52:50 +01:00