Commit Graph

747 Commits

Author SHA1 Message Date
shreyas-goenka 9194418ac1
Fix regex error check in mkdir integration test (#735)
## Changes
Fixes test for all cloud provider after the Go SDK bump which introduces
the `non retryable error` prefix to errors. The test passes now.
2023-09-05 14:25:26 +00:00
Lennart Kats (databricks) e533f9109a
Show 'databricks bundle init' template in CLI prompt (#725)
~(this should be changed to target `main`)~

This reveals the template from
https://github.com/databricks/cli/pull/686 in CLI prompts for once #686
and #708 are merged.

---------

Co-authored-by: Andrew Nester <andrew.nester@databricks.com>
Co-authored-by: PaulCornellDB <paul.cornell@databricks.com>
Co-authored-by: Pieter Noordhuis <pieter.noordhuis@databricks.com>
2023-09-05 13:57:01 +00:00
Lennart Kats (databricks) 8c2cc07f7b
databricks bundle init template v1 (#686)
## Changes

This adds a built-in "default-python" template to the CLI. This is based
on the new default-template support of
https://github.com/databricks/cli/pull/685.

The goal here is to offer an experience where customers can simply type
`databricks bundle init` to get a default template:

```
$ databricks bundle init
Template to use [default-python]: default-python
Unique name for this project [my_project]: my_project
 Successfully initialized template
```

The present template:
- [x] Works well with VS Code
- [x] Works well with the workspace
- [x] Works well with DB Connect
- [x] Uses minimal stubs rather than boiler-plate-heavy examples

I'll have a followup with tests + DLT support.

---------

Co-authored-by: Andrew Nester <andrew.nester@databricks.com>
Co-authored-by: PaulCornellDB <paul.cornell@databricks.com>
Co-authored-by: Pieter Noordhuis <pieter.noordhuis@databricks.com>
2023-09-05 11:58:34 +00:00
Lennart Kats (databricks) 947d5b1e5c
Fix IsServicePrincipal() only working for workspace admins (#732)
## Changes

The latest rendition of isServicePrincipal no longer worked for
non-admin users as it used the "principals get" API.

This new version relies on the property that service principals always
have a UUID as their userName. This was tested with the eng-jaws
principal (8b948b2e-d2b5-4b9e-8274-11b596f3b652).
2023-09-05 11:20:55 +00:00
Pieter Noordhuis f62def3e77
Replace API call to test configuration with dummy authenticate call (#728)
## Changes

This reduces the latency of every workspace command by the duration of a
single API call to retrieve the current user (which can take up to a
full second).

Note: the better place to verify that a request can be authenticated is
the SDK itself.

## Tests

* Unit test to confirm an the empty `*http.Request` can be constructed
* Manually confirmed that the additional API call no longer happens
2023-09-05 11:10:37 +00:00
shreyas-goenka bbbeabf98c
Add support for ordering of input prompts (#662)
## Changes

JSON schema properties are a map and thus unordered.

This PR introduces a JSON schema extension field called `order` to allow
template authors to define the order in which template variables should
be resolved/prompted.

## Tests

Unit tests.

---------

Co-authored-by: Pieter Noordhuis <pieter.noordhuis@databricks.com>
2023-09-05 11:08:25 +00:00
Pieter Noordhuis 2f2386ef5a
Work on GitHub Action (#733)
## Changes

* Run the build workflow on push to main to properly use the build cache

Same as https://github.com/databricks/databricks-sdk-go/pull/601.

## Tests

n/a
2023-09-05 09:58:56 +00:00
Pieter Noordhuis 7a130a3e6e
Group permission related commands (#730)
## Changes

Before:
```
Usage:
  databricks instance-pools [command]

Available Commands:
  create                Create a new instance pool.
  delete                Delete an instance pool.
  edit                  Edit an existing instance pool.
  get                   Get instance pool information.
  get-permission-levels Get instance pool permission levels.
  get-permissions       Get instance pool permissions.
  list                  List instance pool info.
  set-permissions       Set instance pool permissions.
  update-permissions    Update instance pool permissions.
```

After:
```
Usage:
  databricks instance-pools [command]

Available Commands
  create                Create a new instance pool.
  delete                Delete an instance pool.
  edit                  Edit an existing instance pool.
  get                   Get instance pool information.
  list                  List instance pool info.

Permission Commands
  get-permission-levels Get instance pool permission levels.
  get-permissions       Get instance pool permissions.
  set-permissions       Set instance pool permissions.
  update-permissions    Update instance pool permissions.
```

## Tests

Manual.
2023-09-05 09:58:45 +00:00
Pieter Noordhuis 1752e29885
Update Go SDK to v0.19.0 (#729)
## Changes

* Update Go SDK to v0.19.0
* Update commands per OpenAPI spec from Go SDK
* Incorporate `client.Do()` signature change to include a (nil) header
map
* Update `workspace.WorkspaceService` mock with permissions methods
* Skip `files` service in codegen; already implemented under the `fs`
command

## Tests

Unit and integration tests pass.
2023-09-05 09:43:57 +00:00
Pieter Noordhuis 437263eb58
Upgrade to actions/checkout@v4 (#731)
## Changes

This should fix intermittent failures with v3 (see
https://github.com/actions/checkout/issues/1448)
2023-09-05 08:27:18 +00:00
Andrew Nester 83443bae8d
Make resource and artifact paths in bundle config relative to config folder (#708)
# Warning: breaking change

## Changes
Instead of having paths in bundle config files be relative to bundle
root even if the config file is nested, this PR makes such paths
relative to the folder where the config is located.

When bundle is initialised, these paths will be transformed to relative
paths based on bundle root. For example,
we have file structure like this
```
- mybundle
| - bundle.yml
| - subfolder
| -- resource.yml
| -- my.whl
```

Previously, we had to reference `my.whl` in resource.yml like this,
which was confusing because resource.yml is in the same subfolder
```
sync:
  include:
    - ./subfolder/*.whl
...
tasks:
  - task_key: name
    libraries:
      - whl: ./subfolder/my.whl
...
```

After the change we can reference it like this (which is in line with
the current behaviour for notebooks)

```
sync:
  include:
    - ./*.whl
...
tasks:
  - task_key: name
    libraries:
      - whl: ./my.whl
...
```

## Tests
Existing `translate_path_tests` successfully passed after refactoring.

Added a couple of uses cases for `Libraries` paths.

Added a bundle config tests with include config and sync section

---------

Co-authored-by: Pieter Noordhuis <pieter.noordhuis@databricks.com>
2023-09-04 09:55:01 +00:00
Lennart Kats (databricks) e22fd73b7d
Cleanup after previous PR comments (#724)
## Changes

@pietern this addresses a comment from you on a recently merged PR. It
also updates settings.json based on the settings VS Code adds as soon as
you edit a notebook.
2023-09-04 07:07:17 +00:00
Andrew Nester 86c30dd328
Fixed artifact file uploading on Windows and wheel execution on DBR 13.3 (#722)
## Changes
Fixed artifact file uploading on Windows and wheel execution on DBR 13.3

Fixes #719, #720

## Tests
Added regression test for Windows
2023-08-31 14:10:32 +00:00
Pieter Noordhuis cc1038fbd5
Upgrade to actions/setup-go@v4 (#718)
## Changes

Version 4 enables caching by default so we no longer need to explicitly
enable it: https://github.com/actions/setup-go#v4.

The build cache only reuses a cache from a repo's default branch, which
for this repository is `main`. After enabling the merge queue, we no
longer run builds on the `main` branch after push, but on merge queue
branches. With no more builds on the `main` branch there is no longer a
cache to reuse.

This change fixes that by making the `release(-snapshot)?` workflows use
the same caching mechanism. These run off of the `main` branch, so the
cache they save can be reused by builds triggered on PRs or from the
merge queue.

## Tests

We have to merge this to see if it works.
2023-08-30 14:57:34 +00:00
Andrew Nester deebaa89f7
Release v0.203.3 (#716)
Bundles:
* Support cluster overrides with cluster_key and compute_key
([#696](https://github.com/databricks/cli/pull/696)).
* Allow referencing local Python wheels without artifacts section
defined ([#703](https://github.com/databricks/cli/pull/703)).
* Fixed --environment flag
([#705](https://github.com/databricks/cli/pull/705)).
* Correctly identify local paths in libraries section
([#702](https://github.com/databricks/cli/pull/702)).
* Fixed path joining in FindFilesWithSuffixInPath
([#704](https://github.com/databricks/cli/pull/704)).
* Added transformation mutator for Python wheel task for them to work on
DBR <13.1 ([#635](https://github.com/databricks/cli/pull/635)).

Internal:
* Add a foundation for built-in templates
([#685](https://github.com/databricks/cli/pull/685)).
* Test transform when no Python wheel tasks defined
([#714](https://github.com/databricks/cli/pull/714)).
* Pin Terraform binary version to 1.5.5
([#715](https://github.com/databricks/cli/pull/715)).
* Cleanup after "Add a foundation for built-in templates"
([#707](https://github.com/databricks/cli/pull/707)).
* Filter down to Python wheel tasks only for trampoline
([#712](https://github.com/databricks/cli/pull/712)).
* Update Terraform provider schema structs from 1.23.0
([#713](https://github.com/databricks/cli/pull/713)).
2023-08-30 14:31:36 +00:00
Andrew Nester a548eba492
Test transform when no Python wheel tasks defined (#714)
## Changes
Fixed panic from Python transform when no python wheel tasks defined

## Tests
Added regression test
2023-08-30 14:09:15 +00:00
Pieter Noordhuis 46b999ed42
Pin Terraform binary version to 1.5.5 (#715)
## Changes

The installer doesn't respect the version constraints if they are
specified.

Source: [the vc argument is not
used](850464c601/releases/latest_version.go (L158-L177)).

## Tests

Confirmed manually.
2023-08-30 14:08:37 +00:00
Lennart Kats (databricks) 707fd6f617
Cleanup after "Add a foundation for built-in templates" (#707)
## Changes
Add some cleanup based on @pietern's comments on
https://github.com/databricks/cli/pull/685
2023-08-30 14:01:08 +00:00
Pieter Noordhuis aa9e1fc41c
Update Terraform provider schema structs from 1.23.0 (#713)
## Changes

The provider at version 1.24.0 includes a regression for the MLflow
model resource.

To fix this, we explicitly pin the provider version at the version we
generate bindings for.

## Tests

Confirmed that a deploy of said MLflow model resource works with 1.23.0.
2023-08-30 13:58:28 +00:00
Pieter Noordhuis ca2f1dc06c
Filter down to Python wheel tasks only for trampoline (#712)
## Changes

Fixes issue introduced in #635.

## Tests

Added new unit test to confirm correct behavior.

Manually deployed sample bundle.
2023-08-30 13:51:15 +00:00
Andrew Nester 12368e3382
Added transformation mutator for Python wheel task for them to work on DBR <13.1 (#635)
## Changes
***Note: this PR relies on sync.include functionality from here:
https://github.com/databricks/cli/pull/671***

Added transformation mutator for Python wheel task for them to work on
DBR <13.1

Using wheels upload to Workspace file system as cluster libraries is not
supported in DBR < 13.1

In order to make Python wheel work correctly on DBR < 13.1 we do the
following:
1. Build and upload python wheel as usual
2. Transform python wheel task into special notebook task which does the
following
a. Installs all necessary wheels with %pip magic
b. Executes defined entry point with all provided parameters
3. Upload this notebook file to workspace file system
4. Deploy transformed job task

This is also beneficial for executing on existing clusters because this
notebook always reinstall wheels so if there are any changes to the
wheel package, they are correctly picked up

## Tests
bundle.yml

```yaml
bundle:
  name: wheel-task

workspace:
  host: ****

resources:
  jobs:
    test_job:
      name: "[${bundle.environment}] My Wheel Job"
      tasks:
        - task_key: TestTask
          existing_cluster_id: "***"
          python_wheel_task:
            package_name: "my_test_code"
            entry_point: "run"
            parameters: ["first argument","first value","second argument","second value"]
          libraries:
          - whl: ./dist/*.whl
```

Output
```
andrew.nester@HFW9Y94129 wheel % databricks bundle run test_job
Run URL: ***

2023-08-03 15:58:04 "[default] My Wheel Job" TERMINATED SUCCESS 
Output:
=======
Task TestTask:
Hello from my func
Got arguments v1:
['python', 'first argument', 'first value', 'second argument', 'second value']

```
2023-08-30 12:21:39 +00:00
Andrew Nester 3f2cf3c6b7
Fixed path joining in FindFilesWithSuffixInPath (#704)
## Changes
Fixes #693 

## Tests
Newly added tests failed before the fix:
https://github.com/databricks/cli/actions/runs/6000754026/job/16273507998?pr=704
2023-08-29 08:26:26 +00:00
Andrew Nester 842cd8b7ae
Correctly identify local paths in libraries section (#702)
## Changes
Fixes #699

## Tests
Added unit test
2023-08-29 08:26:09 +00:00
Andrew Nester 5477afe4f4
Fixed --environment flag (#705)
## Changes
Fixed --environment flag

Fixes https://github.com/databricks/setup-cli/issues/35

## Tests
Added regression test
2023-08-28 17:05:55 +00:00
Andrew Nester 5f6289e3a7
Allow referencing local Python wheels without artifacts section defined (#703)
## Changes
Now if the user reference local Python wheel files and do not specify
"artifacts" section, this file will be automatically uploaded by CLI.

Fixes #693 

## Tests
Added unit tests

Ran bundle deploy for this configuration
```
resources:
  jobs:
    some_other_job:
      name: "[${bundle.environment}] My Wheel Job"
      tasks:
        - task_key: TestTask
          existing_cluster_id: ${var.job_existing_cluster}
          python_wheel_task:
            package_name: "my_test_code"
            entry_point: "run"
          libraries:
          - whl: ./dist/*.whl
 ```
 
 Result
 
 ```
andrew.nester@HFW9Y94129 wheel % databricks bundle deploy
artifacts.whl.AutoDetect: Detecting Python wheel project...
artifacts.whl.AutoDetect: No Python wheel project found at bundle root folder
Starting upload of bundle files
Uploaded bundle files at /Users/andrew.nester@databricks.com/.bundle/wheel-task/default/files!

artifacts.Upload(my_test_code-0.0.1-py3-none-any.whl): Uploading...
artifacts.Upload(my_test_code-0.0.1-py3-none-any.whl): Upload succeeded
 
 ```
2023-08-28 16:29:04 +00:00
Lennart Kats (databricks) 861f33d376
Support cluster overrides with cluster_key and compute_key (#696)
## Changes

Support `databricks bundle deploy --compute-id my_all_purpose_cluster`
in two missing cases.
2023-08-28 07:51:35 +00:00
Lennart Kats (databricks) a5b86093ec
Add a foundation for built-in templates (#685)
## Changes

This pull request extends the templating support in preparation of a
new, default template (WIP, https://github.com/databricks/cli/pull/686):
* builtin templates that can be initialized using e.g. `databricks
bundle init default-python`
* builtin templates are embedded into the executable using go's `embed`
functionality, making sure they're co-versioned with the CLI
* new helpers to get the workspace name, current user name, etc. help
craft a complete template
* (not enabled yet) when the user types `databricks bundle init` they
can interactively select the `default-python` template

And makes two tangentially related changes:
* IsServicePrincipal now uses the "users" API rather than the
"principals" API, since the latter is too slow for our purposes.
* mode: prod no longer requires the 'target.prod.git' setting. It's hard
to set that from a template. (Pieter is planning an overhaul of warnings
support; this would be one of the first warnings we show.)

The actual `default-python` template is maintained in a separate PR:
https://github.com/databricks/cli/pull/686

## Tests
Unit tests, manual testing
2023-08-25 09:03:42 +00:00
Andrew Nester c5cd20de23
Release v0.203.2 (#694)
CLI:
* Added `databricks account o-auth-enrollment enable` command
([#687](https://github.com/databricks/cli/pull/687)).

Bundles:
* Do not try auto detect Python package if no Python wheel tasks defined
([#674](https://github.com/databricks/cli/pull/674)).
* Renamed `environments` to `targets` in bundle configuration
([#670](https://github.com/databricks/cli/pull/670)).
* Rename init project-dir flag to output-dir
([#676](https://github.com/databricks/cli/pull/676)).
* Added support for sync.include and sync.exclude sections
([#671](https://github.com/databricks/cli/pull/671)).
* Add template directory flag for bundle templates
([#675](https://github.com/databricks/cli/pull/675)).
* Never ignore root directory when enumerating files in a repository
([#683](https://github.com/databricks/cli/pull/683)).
* Improve 'mode' error message
([#681](https://github.com/databricks/cli/pull/681)).
* Added run_as section for bundle configuration
([#692](https://github.com/databricks/cli/pull/692)).
2023-08-24 11:04:20 +00:00
Andrew Nester 4ee926b885
Added run_as section for bundle configuration (#692)
## Changes
Added run_as section for bundle configuration.

This section allows to define an user name or service principal which
will be applied as an execution identity for jobs and DLT pipelines. In
the case of DLT, identity defined in `run_as` will be assigned
`IS_OWNER` permission on this pipeline.

## Tests
Added unit tests for configuration.

Also ran deploy for the following bundle configuration

```
bundle:
  name: "run_as"

run_as:
  # service_principal_name: "f7263fcc-56d0-4981-8baf-c2a45296690b"
  user_name: "lennart.kats@databricks.com"

resources:
  pipelines:
    andrew_pipeline:
      name: "Andrew Nester pipeline"
      libraries:
        - notebook:
            path: ./test.py

  jobs:
    job_one:
      name: Job One
      tasks:
        - task_key: "task"
          new_cluster:
            num_workers: 1
            spark_version: 13.2.x-snapshot-scala2.12
            node_type_id: i3.xlarge
            runtime_engine: PHOTON
          notebook_task: 
            notebook_path: "./test.py"
```
2023-08-23 16:47:07 +00:00
Serge Smertin 5ed635a240
Added `databricks account o-auth-enrollment enable` command (#687)
This command takes the user through the interactive flow to set up OAuth
for a fresh account, where only Basic authentication works.

---------

Co-authored-by: Andrew Nester <andrew.nester@databricks.com>
2023-08-21 16:17:02 +00:00
Lennart Kats (databricks) e1ca24d5d5
Improve 'mode' error message (#681)
## Changes

`unsupported value specified for 'mode': %s` was not a helpful error
message.
2023-08-21 09:09:08 +00:00
Pieter Noordhuis c25bc041b1
Never ignore root directory when enumerating files in a repository (#683)
## Changes

The pattern `.*` in a `.gitignore` file can match `.` when walking all
files in a repository. If it does, then the walker immediately aborts
and no files are returned. The root directory (an unnamed directory)
must never be ignored.

Reported in https://github.com/databricks/databricks-vscode/issues/837.

## Tests

New tests pass.
2023-08-21 07:35:02 +00:00
shreyas-goenka ffc78b4b8b
Add template directory flag for bundle templates (#675)
## Changes
This flag allows users to initialize a template from a subdirectory in
the repo root. Also enables multi template repositories.

## Tests
Manually
2023-08-18 09:29:48 +00:00
Andrew Nester e3e9bc6def
Added support for sync.include and sync.exclude sections (#671)
## Changes
Added support for `sync.include` and `sync.exclude` sections

## Tests
Added `sample-java` folder to gitignore

```
bundle:
  name: wheel-task

sync:
  include:
    - "./sample-java/*.kts"
```

Kotlin files were correctly synced.

```
[DEBUG] Test execution command:  /opt/homebrew/opt/go@1.21/bin/go test ./... -json -timeout 1h -coverpkg=./... -coverprofile=coverage.txt -run ^TestAcc
[DEBUG] Test execution directory:  /Users/andrew.nester/cli
2023/08/17 17:12:10 [INFO]  TestAccAlertsCreateErrWhenNoArguments (2.320s)
2023/08/17 17:12:10 [INFO]  TestAccApiGet (0.650s)
2023/08/17 17:12:12 [INFO]  TestAccClustersList (1.060s)
2023/08/17 17:12:12 [INFO]  TestAccClustersGet (0.760s)
2023/08/17 17:12:26 [INFO]  TestAccFilerWorkspaceFilesReadWrite (13.270s)
2023/08/17 17:12:32 [INFO]  TestAccFilerWorkspaceFilesReadDir (6.860s)
2023/08/17 17:12:46 [INFO]  TestAccFilerDbfsReadWrite (13.380s)
2023/08/17 17:12:53 [INFO]  TestAccFilerDbfsReadDir (7.460s)
2023/08/17 17:13:01 [INFO]  TestAccFilerWorkspaceNotebookConflict (7.920s)
2023/08/17 17:13:10 [INFO]  TestAccFilerWorkspaceNotebookWithOverwriteFlag (9.290s)
2023/08/17 17:13:10 [INFO]  TestAccFilerLocalReadWrite (0.010s)
2023/08/17 17:13:11 [INFO]  TestAccFilerLocalReadDir (0.010s)
2023/08/17 17:13:14 [INFO]  TestAccFsCatForDbfs (3.180s)
2023/08/17 17:13:15 [INFO]  TestAccFsCatForDbfsOnNonExistentFile (0.940s)
2023/08/17 17:13:15 [INFO]  TestAccFsCatForDbfsInvalidScheme (0.560s)
2023/08/17 17:13:18 [INFO]  TestAccFsCatDoesNotSupportOutputModeJson (2.910s)
2023/08/17 17:13:51 [INFO]  TestAccFsCpDir (32.730s)
2023/08/17 17:14:06 [INFO]  TestAccFsCpFileToFile (14.740s)
2023/08/17 17:14:20 [INFO]  TestAccFsCpFileToDir (14.340s)
2023/08/17 17:14:53 [INFO]  TestAccFsCpDirToDirFileNotOverwritten (32.710s)
2023/08/17 17:15:12 [INFO]  TestAccFsCpFileToDirFileNotOverwritten (19.590s)
2023/08/17 17:15:32 [INFO]  TestAccFsCpFileToFileFileNotOverwritten (19.950s)
2023/08/17 17:16:11 [INFO]  TestAccFsCpDirToDirWithOverwriteFlag (38.970s)
2023/08/17 17:16:32 [INFO]  TestAccFsCpFileToFileWithOverwriteFlag (21.040s)
2023/08/17 17:16:52 [INFO]  TestAccFsCpFileToDirWithOverwriteFlag (19.670s)
2023/08/17 17:16:54 [INFO]  TestAccFsCpErrorsWhenSourceIsDirWithoutRecursiveFlag (1.890s)
2023/08/17 17:16:54 [INFO]  TestAccFsCpErrorsOnInvalidScheme (0.690s)
2023/08/17 17:17:10 [INFO]  TestAccFsCpSourceIsDirectoryButTargetIsFile (15.810s)
2023/08/17 17:17:14 [INFO]  TestAccFsLsForDbfs (4.000s)
2023/08/17 17:17:18 [INFO]  TestAccFsLsForDbfsWithAbsolutePaths (4.000s)
2023/08/17 17:17:21 [INFO]  TestAccFsLsForDbfsOnFile (3.140s)
2023/08/17 17:17:23 [INFO]  TestAccFsLsForDbfsOnEmptyDir (2.030s)
2023/08/17 17:17:24 [INFO]  TestAccFsLsForDbfsForNonexistingDir (0.840s)
2023/08/17 17:17:25 [INFO]  TestAccFsLsWithoutScheme (0.590s)
2023/08/17 17:17:27 [INFO]  TestAccFsMkdirCreatesDirectory (2.310s)
2023/08/17 17:17:30 [INFO]  TestAccFsMkdirCreatesMultipleDirectories (2.800s)
2023/08/17 17:17:33 [INFO]  TestAccFsMkdirWhenDirectoryAlreadyExists (2.700s)
2023/08/17 17:17:35 [INFO]  TestAccFsMkdirWhenFileExistsAtPath (2.870s)
2023/08/17 17:17:40 [INFO]  TestAccFsRmForFile (4.030s)
2023/08/17 17:17:43 [INFO]  TestAccFsRmForEmptyDirectory (3.470s)
2023/08/17 17:17:46 [INFO]  TestAccFsRmForNonEmptyDirectory (3.350s)
2023/08/17 17:17:47 [INFO]  TestAccFsRmForNonExistentFile (0.940s)
2023/08/17 17:17:51 [INFO]  TestAccFsRmForNonEmptyDirectoryWithRecursiveFlag (3.570s)
2023/08/17 17:17:52 [INFO]  TestAccGitClone (0.890s)
2023/08/17 17:17:52 [INFO]  TestAccGitCloneWithOnlyRepoNameOnAlternateBranch (0.730s)
2023/08/17 17:17:53 [INFO]  TestAccGitCloneErrorsWhenRepositoryDoesNotExist (0.540s)
2023/08/17 17:18:02 [INFO]  TestAccLock (8.800s)
2023/08/17 17:18:06 [INFO]  TestAccLockUnlockWithoutAllowsLockFileNotExist (3.930s)
2023/08/17 17:18:09 [INFO]  TestAccLockUnlockWithAllowsLockFileNotExist (3.320s)
2023/08/17 17:18:20 [INFO]  TestAccSyncFullFileSync (10.570s)
2023/08/17 17:18:31 [INFO]  TestAccSyncIncrementalFileSync (11.460s)
2023/08/17 17:18:42 [INFO]  TestAccSyncNestedFolderSync (10.850s)
2023/08/17 17:18:53 [INFO]  TestAccSyncNestedFolderDoesntFailOnNonEmptyDirectory (10.650s)
2023/08/17 17:19:04 [INFO]  TestAccSyncNestedSpacePlusAndHashAreEscapedSync (10.930s)
2023/08/17 17:19:11 [INFO]  TestAccSyncIncrementalFileOverwritesFolder (7.010s)
2023/08/17 17:19:18 [INFO]  TestAccSyncIncrementalSyncPythonNotebookToFile (7.380s)
2023/08/17 17:19:24 [INFO]  TestAccSyncIncrementalSyncFileToPythonNotebook (6.220s)
2023/08/17 17:19:30 [INFO]  TestAccSyncIncrementalSyncPythonNotebookDelete (5.530s)
2023/08/17 17:19:32 [INFO]  TestAccSyncEnsureRemotePathIsUsableIfRepoDoesntExist (2.620s)
2023/08/17 17:19:38 [INFO]  TestAccSyncEnsureRemotePathIsUsableIfRepoExists (5.460s)
2023/08/17 17:19:40 [INFO]  TestAccSyncEnsureRemotePathIsUsableInWorkspace (1.850s)
2023/08/17 17:19:40 [INFO]  TestAccWorkspaceList (0.780s)
2023/08/17 17:19:51 [INFO]  TestAccExportDir (10.350s)
2023/08/17 17:19:54 [INFO]  TestAccExportDirDoesNotOverwrite (3.330s)
2023/08/17 17:19:58 [INFO]  TestAccExportDirWithOverwriteFlag (3.770s)
2023/08/17 17:20:07 [INFO]  TestAccImportDir (9.320s)
2023/08/17 17:20:24 [INFO]  TestAccImportDirDoesNotOverwrite (16.950s)
2023/08/17 17:20:35 [INFO]  TestAccImportDirWithOverwriteFlag (10.620s)
2023/08/17 17:20:35 [INFO]  68/68 passed, 0 failed, 3 skipped
```
2023-08-18 08:07:25 +00:00
shreyas-goenka 042fbaa614
Rename init project-dir flag to output-dir (#676)
## Changes
This PR:
1. Renames the project-dir flag to output-dir
2. Makes the project dir flag optional. When unspecified we default to
the current working directory.

## Tests
Manually

---------

Co-authored-by: Pieter Noordhuis <pieter.noordhuis@databricks.com>
2023-08-17 20:32:30 +00:00
Andrew Nester 56dcd3f0a7
Renamed `environments` to `targets` in bundle configuration (#670)
## Changes
Renamed Environments to Targets in bundle.yml.

The change is backward-compatible and customers can continue to use
`environments` in the time being.

## Tests
Added tests which checks that both `environments` and `targets` sections
in bundle.yml works correctly
2023-08-17 15:22:32 +00:00
Andrew Nester 4694832534
Do not try auto detect Python package if no Python wheel tasks defined (#674)
## Changes
Fixes #673 

It also includes a change for `libraries` from #635 to get the list of
wheel tasks
2023-08-17 09:11:39 +00:00
Pieter Noordhuis 35e8ed30c6
Release v0.203.1 (#672)
CLI:
* Always resolve .databrickscfg file
([#659](https://github.com/databricks/cli/pull/659)).

Bundles:
* Add internal tag for bundle fields to be skipped from schema
([#636](https://github.com/databricks/cli/pull/636)).
* Log the bundle root configuration file if applicable
([#657](https://github.com/databricks/cli/pull/657)).
* Execute paths without the .tmpl extension as templates
([#654](https://github.com/databricks/cli/pull/654)).
* Enable environment overrides for job clusters
([#658](https://github.com/databricks/cli/pull/658)).
* Merge artifacts and resources block with overrides enabled
([#660](https://github.com/databricks/cli/pull/660)).
* Locked terraform binary version to <= 1.5.5
([#666](https://github.com/databricks/cli/pull/666)).
* Return better error messages for invalid JSON schema types in
templates ([#661](https://github.com/databricks/cli/pull/661)).
* Use custom prompter for bundle template inputs
([#663](https://github.com/databricks/cli/pull/663)).
* Add map and pair helper functions for bundle templates
([#665](https://github.com/databricks/cli/pull/665)).
* Correct name for force acquire deploy flag
([#656](https://github.com/databricks/cli/pull/656)).
* Confirm that override with a zero value doesn't work
([#669](https://github.com/databricks/cli/pull/669)).

Internal:
* Consolidate functions in libs/git
([#652](https://github.com/databricks/cli/pull/652)).
* Upgraded Go version to 1.21
([#664](https://github.com/databricks/cli/pull/664)).
2023-08-16 12:56:12 +00:00
Pieter Noordhuis d225d7a662
Confirm that override with a zero value doesn't work (#669)
## Changes

This is not desirable and will be addressed by representing our
configuration in a different structure (e.g. with cty, or with
plain `any`), instead of Go structs.

## Tests

Pass.
2023-08-16 11:28:57 +00:00
shreyas-goenka 6a843f28ef
Correct name for force acquire deploy flag (#656)
## Changes
As discussed here, the name for this flag should be `force-lock`:
https://github.com/databricks/cli/pull/578#discussion_r1276233445

## Tests
Manually and existing tests
2023-08-15 19:03:43 +00:00
shreyas-goenka 6c644e159c
Add map and pair helper functions for bundle templates (#665)
## Changes
Go text templates allows only specifying one input argument for
invocations of associated templates (ie `{{template ...}}`). This PR
introduces the map and pair functions which allow template authors to
work around this limitation by passing multiple arguments as key value
pairs in a map.

This PR is based on feedback from the mlops stacks migration where
otherwise a bunch of duplicate code is required for computed values and
fixtures.

## Tests
Unit test
2023-08-15 16:07:22 +00:00
shreyas-goenka 61b103318f
Use custom prompter for bundle template inputs (#663)
## Changes
Prompt UI glitches often. We are switching to a custom implementation of
a simple prompter which is much more stable.
This also allows new lines in prompts which has been an ask by the
mlflow team.

## Tests
Tested manually
2023-08-15 14:50:20 +00:00
shreyas-goenka 878bb6deae
Return better error messages for invalid JSON schema types in templates (#661)
## Changes
Adds a function to validate json schema types added by the author. The
default json unmarshaller does not validate that the parsed type matches
the enum defined in `jsonschema.Type`

Includes some other improvements to provide better error messages.

This PR was prompted by usability difficulties reported by @mingyu89
during mlops stack migration.

## Tests
Unit tests
2023-08-15 14:28:04 +00:00
Andrew Nester 6e708da6fc
Upgraded Go version to 1.21 (#664)
## Changes
Upgraded Go version to 1.21

Upgraded to use `slices` and `slog` from core instead of experimental.

Still use `exp/maps` as our code relies on `maps.Keys` which is not part
of core package and therefore refactoring required.

### Tests

Integration tests passed

```
[DEBUG] Test execution command:  /opt/homebrew/opt/go@1.21/bin/go test ./... -json -timeout 1h -run ^TestAcc
[DEBUG] Test execution directory:  /Users/andrew.nester/cli
2023/08/15 13:20:51 [INFO]  TestAccAlertsCreateErrWhenNoArguments (2.150s)
2023/08/15 13:20:52 [INFO]  TestAccApiGet (0.580s)
2023/08/15 13:20:53 [INFO]  TestAccClustersList (0.900s)
2023/08/15 13:20:54 [INFO]  TestAccClustersGet (0.870s)
2023/08/15 13:21:06 [INFO]  TestAccFilerWorkspaceFilesReadWrite (11.980s)
2023/08/15 13:21:13 [INFO]  TestAccFilerWorkspaceFilesReadDir (7.060s)
2023/08/15 13:21:25 [INFO]  TestAccFilerDbfsReadWrite (12.810s)
2023/08/15 13:21:33 [INFO]  TestAccFilerDbfsReadDir (7.380s)
2023/08/15 13:21:41 [INFO]  TestAccFilerWorkspaceNotebookConflict (7.760s)
2023/08/15 13:21:49 [INFO]  TestAccFilerWorkspaceNotebookWithOverwriteFlag (8.660s)
2023/08/15 13:21:49 [INFO]  TestAccFilerLocalReadWrite (0.020s)
2023/08/15 13:21:49 [INFO]  TestAccFilerLocalReadDir (0.010s)
2023/08/15 13:21:52 [INFO]  TestAccFsCatForDbfs (3.190s)
2023/08/15 13:21:53 [INFO]  TestAccFsCatForDbfsOnNonExistentFile (0.890s)
2023/08/15 13:21:54 [INFO]  TestAccFsCatForDbfsInvalidScheme (0.600s)
2023/08/15 13:21:57 [INFO]  TestAccFsCatDoesNotSupportOutputModeJson (2.960s)
2023/08/15 13:22:28 [INFO]  TestAccFsCpDir (31.480s)
2023/08/15 13:22:43 [INFO]  TestAccFsCpFileToFile (14.530s)
2023/08/15 13:22:58 [INFO]  TestAccFsCpFileToDir (14.610s)
2023/08/15 13:23:29 [INFO]  TestAccFsCpDirToDirFileNotOverwritten (31.810s)
2023/08/15 13:23:47 [INFO]  TestAccFsCpFileToDirFileNotOverwritten (17.500s)
2023/08/15 13:24:04 [INFO]  TestAccFsCpFileToFileFileNotOverwritten (17.260s)
2023/08/15 13:24:37 [INFO]  TestAccFsCpDirToDirWithOverwriteFlag (32.690s)
2023/08/15 13:24:56 [INFO]  TestAccFsCpFileToFileWithOverwriteFlag (19.290s)
2023/08/15 13:25:15 [INFO]  TestAccFsCpFileToDirWithOverwriteFlag (19.230s)
2023/08/15 13:25:17 [INFO]  TestAccFsCpErrorsWhenSourceIsDirWithoutRecursiveFlag (2.010s)
2023/08/15 13:25:18 [INFO]  TestAccFsCpErrorsOnInvalidScheme (0.610s)
2023/08/15 13:25:33 [INFO]  TestAccFsCpSourceIsDirectoryButTargetIsFile (14.900s)
2023/08/15 13:25:37 [INFO]  TestAccFsLsForDbfs (3.770s)
2023/08/15 13:25:41 [INFO]  TestAccFsLsForDbfsWithAbsolutePaths (4.160s)
2023/08/15 13:25:44 [INFO]  TestAccFsLsForDbfsOnFile (2.990s)
2023/08/15 13:25:46 [INFO]  TestAccFsLsForDbfsOnEmptyDir (1.870s)
2023/08/15 13:25:46 [INFO]  TestAccFsLsForDbfsForNonexistingDir (0.850s)
2023/08/15 13:25:47 [INFO]  TestAccFsLsWithoutScheme (0.560s)
2023/08/15 13:25:49 [INFO]  TestAccFsMkdirCreatesDirectory (2.310s)
2023/08/15 13:25:52 [INFO]  TestAccFsMkdirCreatesMultipleDirectories (2.920s)
2023/08/15 13:25:55 [INFO]  TestAccFsMkdirWhenDirectoryAlreadyExists (2.320s)
2023/08/15 13:25:57 [INFO]  TestAccFsMkdirWhenFileExistsAtPath (2.820s)
2023/08/15 13:26:01 [INFO]  TestAccFsRmForFile (4.030s)
2023/08/15 13:26:05 [INFO]  TestAccFsRmForEmptyDirectory (3.530s)
2023/08/15 13:26:08 [INFO]  TestAccFsRmForNonEmptyDirectory (3.190s)
2023/08/15 13:26:09 [INFO]  TestAccFsRmForNonExistentFile (0.830s)
2023/08/15 13:26:13 [INFO]  TestAccFsRmForNonEmptyDirectoryWithRecursiveFlag (3.580s)
2023/08/15 13:26:13 [INFO]  TestAccGitClone (0.800s)
2023/08/15 13:26:14 [INFO]  TestAccGitCloneWithOnlyRepoNameOnAlternateBranch (0.790s)
2023/08/15 13:26:15 [INFO]  TestAccGitCloneErrorsWhenRepositoryDoesNotExist (0.540s)
2023/08/15 13:26:23 [INFO]  TestAccLock (8.630s)
2023/08/15 13:26:27 [INFO]  TestAccLockUnlockWithoutAllowsLockFileNotExist (3.490s)
2023/08/15 13:26:30 [INFO]  TestAccLockUnlockWithAllowsLockFileNotExist (3.130s)
2023/08/15 13:26:39 [INFO]  TestAccSyncFullFileSync (9.370s)
2023/08/15 13:26:50 [INFO]  TestAccSyncIncrementalFileSync (10.390s)
2023/08/15 13:27:00 [INFO]  TestAccSyncNestedFolderSync (10.680s)
2023/08/15 13:27:11 [INFO]  TestAccSyncNestedFolderDoesntFailOnNonEmptyDirectory (10.970s)
2023/08/15 13:27:22 [INFO]  TestAccSyncNestedSpacePlusAndHashAreEscapedSync (10.930s)
2023/08/15 13:27:29 [INFO]  TestAccSyncIncrementalFileOverwritesFolder (7.020s)
2023/08/15 13:27:37 [INFO]  TestAccSyncIncrementalSyncPythonNotebookToFile (7.380s)
2023/08/15 13:27:43 [INFO]  TestAccSyncIncrementalSyncFileToPythonNotebook (6.050s)
2023/08/15 13:27:48 [INFO]  TestAccSyncIncrementalSyncPythonNotebookDelete (5.390s)
2023/08/15 13:27:51 [INFO]  TestAccSyncEnsureRemotePathIsUsableIfRepoDoesntExist (2.570s)
2023/08/15 13:27:56 [INFO]  TestAccSyncEnsureRemotePathIsUsableIfRepoExists (5.540s)
2023/08/15 13:27:58 [INFO]  TestAccSyncEnsureRemotePathIsUsableInWorkspace (1.840s)
2023/08/15 13:27:59 [INFO]  TestAccWorkspaceList (0.790s)
2023/08/15 13:28:08 [INFO]  TestAccExportDir (8.860s)
2023/08/15 13:28:11 [INFO]  TestAccExportDirDoesNotOverwrite (3.090s)
2023/08/15 13:28:14 [INFO]  TestAccExportDirWithOverwriteFlag (3.500s)
2023/08/15 13:28:23 [INFO]  TestAccImportDir (8.330s)
2023/08/15 13:28:34 [INFO]  TestAccImportDirDoesNotOverwrite (10.970s)
2023/08/15 13:28:44 [INFO]  TestAccImportDirWithOverwriteFlag (10.130s)
2023/08/15 13:28:44 [INFO]  68/68 passed, 0 failed, 3 skipped
```
2023-08-15 13:50:40 +00:00
Andrew Nester 5cdaacacc3
Locked terraform binary version to <= 1.5.5 (#666)
## Changes
Locked terraform binary version to <= 1.5.5
2023-08-15 13:39:32 +00:00
Pieter Noordhuis 8dc6936581
Merge artifacts and resources block with overrides enabled (#660)
## Changes

Originally, these blocks were merged with overrides. This was
(inadvertently) disabled in #94. This change re-enables merging these
blocks with overrides, such that any field set in an environment
override always takes precedence over the field set in the base
definition.

## Tests

New unit test passes.
2023-08-15 09:58:54 +00:00
Miles Yucht 5b819cd982
Always resolve .databrickscfg file (#659)
## Changes
#629 introduced a change to autopopulate the host from .databrickscfg if
the user is logging back into a host they were previously using. This
did not respect the DATABRICKS_CONFIG_FILE env variable, causing the
flow to stop working for users with no .databrickscfg file in their home
directory.

This PR refactors all config file loading to go through one interface,
`databrickscfg.GetDatabricksCfg()`, and an auxiliary
`databrickscfg.GetDatabricksCfgPath()` to get the configured file path.

Closes #655.

## Tests
```
$ databricks auth login --profile abc
Error: open /Users/miles/.databrickscfg: no such file or directory

$ ./cli auth login --profile abc
Error: cannot load Databricks config file: open /Users/miles/.databrickscfg: no such file or directory


$ DATABRICKS_CONFIG_FILE=~/.databrickscfg.bak ./cli auth login --profile abc
Databricks Host: https://asdf
```
2023-08-14 12:45:08 +00:00
Pieter Noordhuis 97699b849f
Enable environment overrides for job clusters (#658)
## Changes

While they are a slice, we can identify a job cluster by its job cluster
key. A job definition with multiple job clusters with the same key is
always invalid. We can therefore merge definitions with the same key
into one. This is compatible with how environment overrides are applied;
merging a slice means appending to it. The override will end up in the
job cluster slice of the original, which gives us a deterministic way to
merge them.

Since the alternative is an invalid configuration, this doesn't change
behavior.

## Tests

New test coverage.
2023-08-14 06:43:45 +00:00
shreyas-goenka 6ea70c82a9
Execute paths without the .tmpl extension as templates (#654)
## Changes
The `.tmpl` extension is only meant as a qualifier for whether the file
content is executed as a template. All file paths in the `template`
directory should be treated as valid go text templates.

Before only paths with the `.tmpl` extensions would be resolved as
templates, after this change, all file paths are interpreted as
templates.

## Tests
Unit test. The newly added unit tests also asserts that the file path is
correct, even when the `.tmpl` extension is missing.
2023-08-11 13:48:32 +00:00
Pieter Noordhuis 8656c4a1fa
Log the bundle root configuration file if applicable (#657)
## Changes

Pass through the `context.Context` to the bundle loader functions.

## Tests

Unit tests pass.
2023-08-11 12:28:05 +00:00