steve/databricks-cli
steve
/
databricks-cli
mirror of https://github.com/databricks/cli.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
1,373 Commits 113 Branches 106 Tags 58 MiB
Commit Graph

551 Commits

Author SHA1 Message Date
Ilya Kuznetsov 66bf5f35c1
fix: Use warning instead of log in python wheel wrapper message 2024-11-20 11:48:52 +01:00
Ilya Kuznetsov 6a036d199a
Merge branch 'tmp/in-place-prototype' of github.com:databricks/cli into tmp/in-place-prototype 2024-11-20 11:39:46 +01:00
Ilya Kuznetsov eede522d87
Update bundle/config/presets.go 
Co-authored-by: Pieter Noordhuis <pieter.noordhuis@databricks.com>
 2024-11-20 11:34:35 +01:00
Ilya Kuznetsov ddb68a76f2
fix: Update skip uploading log entry with more clear message 2024-11-20 11:33:09 +01:00
Ilya Kuznetsov aba35ae3fe fix: Replaced diag warning message with logger and removed unnecessary test 2024-11-19 22:08:45 +01:00
Ilya Kuznetsov 530a2a9bab
fix: Skipping path translation tests for windows 2024-11-19 13:33:02 +01:00
Ilya Kuznetsov 5d04569112
fix: Windows tests 2024-11-19 13:15:52 +01:00
Ilya Kuznetsov 8ee8de554b
test: Added path translation test cases for source-linked 2024-11-19 12:46:31 +01:00
Ilya Kuznetsov 5d09070fcb
fix: Cleanup 2024-11-19 11:50:28 +01:00
Ilya Kuznetsov 234d971790
fix: Cleanup in test 2024-11-19 11:47:57 +01:00
Ilya Kuznetsov 8c8fb35861
fix: Add explicit warning when using python wheel wrappers with source-linked deployment 2024-11-19 10:46:40 +01:00
Ilya Kuznetsov aeb9813d25
feat: Use path translations instead of overriding config 2024-11-19 10:40:16 +01:00
Ilya Kuznetsov 518aa14b64
Revert "fix: Skip permissions set and check for in-place" 
This reverts commit 53e1f6df6a.
 2024-11-18 22:32:36 +01:00
Ilya Kuznetsov 55f715dddf
fix: Broken test 2024-11-18 17:31:52 +01:00
Ilya Kuznetsov 1d7b27e0ef
feat: Rename "in-place" to "source-linked" 2024-11-18 17:21:33 +01:00
Ilya Kuznetsov e8825d559f
fix: Reuse existing `root` variable 2024-11-18 16:19:26 +01:00
Ilya Kuznetsov 00bb683be9
fix: Remove unnecessary fields in apply presets test 2024-11-18 16:13:14 +01:00
Ilya Kuznetsov 53e1f6df6a
fix: Skip permissions set and check for in-place 2024-11-18 16:05:21 +01:00
Ilya Kuznetsov e65b50cf3c
fix: Wrong condition 2024-11-18 12:22:14 +01:00
Ilya Kuznetsov 49f6bc9354
fix: Skipping in-place tests on Windows 2024-11-18 12:08:10 +01:00
Ilya Kuznetsov e7165ec17a
feat: Add Databricks Workspace conditions to dev mode setting. Add warning if conditions not met when preset is enabled by user 2024-11-18 11:58:56 +01:00
Ilya Kuznetsov 80ea3a05e3
fix: Move SyncRoot field to the top og the struct 2024-11-18 11:24:33 +01:00
Ilya Kuznetsov a3c6d571c8
fix: Use SyncRootPath 2024-11-18 11:11:44 +01:00
Ilya Kuznetsov e9b72895dc
test: Process target mode 2024-11-15 12:32:44 +01:00
Ilya Kuznetsov 8cd95eb0ea
test: Apply presets unit test 2024-11-15 12:09:53 +01:00
Ilya Kuznetsov 4cf692924c
fix: Missing root variable 2024-11-14 18:46:10 +01:00
Ilya Kuznetsov 51c8ef592c
fix: Applies autoformat 2024-11-14 18:38:31 +01:00
Ilya Kuznetsov 26f24538ee feat: Use dbr package for runtime check 2024-11-14 18:18:17 +01:00
Ilya Kuznetsov 0004ed27a7 Merge branch 'main' of github.com:databricks/cli into tmp/in-place-prototype 2024-11-14 18:12:34 +01:00
Pieter Noordhuis 1508d65c4c
Extract functionality to detect if the CLI is running on DBR (#1889) 
## Changes

Whether or not the CLI is running on DBR can be detected once and stored
in the command's context.

By storing it in the context, it can easily be mocked for testing.

This builds on the simpler approach and conversation in #1744. It
unblocks testing of the DBR-specific paths while not compromising on the
checks we can perform to test if the CLI is running on DBR.

## Tests

* Unit tests for the new `dbr` package
* New unit test for the `ConfigureWSFS` mutator
 2024-11-14 16:10:45 +00:00
Pieter Noordhuis 21d27885dc
Upgrade TF provider to 1.58.0 (#1900) 
## Changes

Notable changes:
* Adds support for `restart_window` for pipelines.
* Fix drift for pipelines where `catalog` contains uppercase characters.
* Better error message if single-node job clusters are incorrectly configured.

See:
* https://github.com/databricks/terraform-provider-databricks/releases/tag/v1.58.0
* https://github.com/databricks/terraform-provider-databricks/releases/tag/v1.57.0
* https://github.com/databricks/terraform-provider-databricks/releases/tag/v1.56.0
* https://github.com/databricks/terraform-provider-databricks/releases/tag/v1.55.0

## Tests

Integration tests pass.
 2024-11-14 14:00:15 +01:00
dependabot[bot] 25838ee0af
Bump github.com/databricks/databricks-sdk-go from 0.49.0 to 0.51.0 (#1878) 
Known issues:

- [ ] _(non-blocking with a command override)_ `apps.Update` requires 2
`name` params (one from path, one from request body)
- [ ] _(non-blocking)_ `lakeview.Create` does not require positional
argument `display_name` anymore because it's not marked as required in
request body

Bumps
[github.com/databricks/databricks-sdk-go](https://github.com/databricks/databricks-sdk-go)
from 0.49.0 to 0.51.0.

---------

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Andrew Nester <andrew.nester@databricks.com>
 2024-11-13 13:40:53 +00:00
Ilya Kuznetsov b2164c0725
fix: Use SyncRootPath to allow using parent directories 2024-11-12 15:16:47 +01:00
Ilya Kuznetsov 5a22151580 feat: Move FilePath update to apply preset step 2024-11-12 09:58:36 +01:00
Ilya Kuznetsov a43f7ddad0 feat: Use global FilePath instead of locally overriden paths 2024-11-11 19:36:31 +01:00
Ilya Kuznetsov fbb9be9bdc
fix: Add SyncRoot mock to tests 2024-11-05 19:03:27 +01:00
Ilya Kuznetsov ae26beabfe Merge branch 'main' of github.com:databricks/cli into tmp/in-place-prototype 2024-11-05 18:13:27 +01:00
Ilya Kuznetsov 6b22fdebba Prototype for in-place deployments in DEV mode 2024-11-05 18:12:45 +01:00
Pieter Noordhuis 26afab2ccb
Fix relative path resolution for dashboards on Windows (#1881) 
## Changes

The file presence check for dashboard files was missing a
`filepath.ToSlash`.

This means it didn't work on Windows unless the dashboard was located at
a path without slashes (i.e. the bundle root).

Closes #1875.

## Tests

* Added a unit test to cover this case (failed before the fix).
* Manually ran a dashboard deployment on Windows.
 2024-11-05 09:53:53 +00:00
Andrew Nester ac71d2e5ce
Fixed adding /Workspace prefix for resource paths (#1866) 
## Changes
`/Workspace` prefix needs to be added to `resource_path` as well.

Fixes the issue mentioned here:
https://github.com/databricks/cli/pull/1822#issuecomment-2447697498

Fixes #1867 

## Tests
Added regression test
 2024-10-30 17:34:11 +00:00
Andrew Nester f018daf413
Use SetPermissions instead of UpdatePermissions when setting folder permissions based on top-level ones (#1822) 
## Changes
Changed to use SetPermissions() to configure the permissions which
remove other permissions on deployment folders.

## Tests
Added unit test
 2024-10-29 12:06:38 +00:00
Pieter Noordhuis 1896b09350
Add bundle generate variant for dashboards (#1847) 
## Changes

This change adds the `databricks bundle generate dashboard` command.

The command requires one of three flags:
* `--existing-id` to generate configuration for an existing dashboard by
its ID.
* `--existing-path` to generate configuration for an existing dashboard
by its path in the workspace file system.
* `--resource` to generate the `.lvdash.json` dashboard file for a
dashboard that's already defined in the bundle. This option does not
impact the YAML configuration.

A typical workflow could look like this:
1. Use the command with `--existing-id` or `--existing-path` for a
starting point
2. Run `bundle deploy` to deploy a copy of the dashboard
3. Run `bundle open` to open this copy in your browser
4. Navigate to the draft mode and make modifications
5. Run `bundle generate dashboard` with `--resource` to update the local
`.lvdash.json` file with the remote modifications

## Tests

* Unit tests.
* Manual walkthrough as documented in the [Dashboard for NYC Taxi Trip
Analysis
example](https://github.com/databricks/bundle-examples/tree/main/knowledge_base/dashboard_nyc_taxi).
 2024-10-29 11:51:59 +00:00
Pieter Noordhuis 11f75fd320
Add support for AI/BI dashboards (#1743) 
## Changes

This change adds support for modeling [AI/BI dashboards][docs] in DABs.


[Example bundle configuration][example] is located in the
`bundle-examples` repository.

[docs]: https://docs.databricks.com/en/dashboards/index.html#dashboards
[example]:
https://github.com/databricks/bundle-examples/tree/main/knowledge_base/dashboard_nyc_taxi

## Tests

* Added unit tests for self-contained parts
* Integration test for e2e dashboard deployment and remote change
modification
 2024-10-29 09:11:08 +00:00
Pieter Noordhuis ed84a33b0a
Reuse resource resolution code for the run command (#1858) 
## Changes

As of #1846 we have a generalized package for doing resource lookups and
completion.

This change updates the run command to use this instead of more specific
code under `bundle/run`.

## Tests

* Unit tests pass
* Manually confirmed that completion and prompting works
 2024-10-24 13:24:30 +00:00
Andrew Nester eaea308254
Added validator for folder permissions (#1824) 
## Changes
This validator checks permissions defined in top-level bundle config and
permissions set in workspace for the folders bundle is deployed to. It
raises the warning if the permissions defined in the workspace are not
defined in bundle.

This validator is executed only during `bundle validate` command.

## Tests

```
Warning: untracked permissions apply to target workspace path

The following permissions apply to the workspace folder at "/Workspace/Users/andrew.nester@databricks.com/.bundle/clusters/default" but are not configured in the bundle:
- level: CAN_MANAGE, user_name: andrew.nester@databricks.com
```

---------

Co-authored-by: Pieter Noordhuis <pieter.noordhuis@databricks.com>
 2024-10-24 12:36:17 +00:00
Pieter Noordhuis 89ee7d8a99
Add command to open a resource in the browser (#1846) 
## Changes

This builds on the functionality added in #1731 that produces a URL for
every resource.

Adds `bundle/resources` package to deal with resource lookups and
command completion. The new functionality is similar to the lookup and
command completion functionality located in `bundle/run`. It differs in
that it doesn't gracefully deal with ambiguous references to resources,
now that we explicitly validate this doesn't occur in the bundle
configuration. It still allows resources to be looked up with their
fully qualified key, `<plural type>.<key>`.

## Tests

* Added unit tests for resource lookup and completion
* Manually confirmed that `bundle open` prompts, accepts a key argument,
and opens a browser
 2024-10-24 12:20:33 +00:00
shreyas-goenka 3bab21e72e
Fix race condition when restarting continuous jobs (#1849) 
## Changes
We don't need to cancel existing runs when the job is continuous and
unpaused. The `/jobs/run-now` command will cancel the existing run and
trigger a new one automatically.

Cancelling the job manually can cause a race condition where both the
manual trigger from the CLI and the continuous trigger from the job
configuration happens at the same time. This PR prevents that from
happening.

## Tests
Unit tests and manually
 2024-10-22 14:59:17 +00:00
Andrew Nester 68d69d6e0b
Upgrade TF provider to 1.54.0 (#1852) 
## Changes
Upgrade TF provider to 1.54.0
 2024-10-22 10:43:43 +00:00
dependabot[bot] f8bb3a8d72
Bump github.com/databricks/databricks-sdk-go from 0.48.0 to 0.49.0 (#1843) 
Bumps
[github.com/databricks/databricks-sdk-go](https://github.com/databricks/databricks-sdk-go)
from 0.48.0 to 0.49.0.
<details>
<summary>Release notes</summary>
<p><em>Sourced from <a
href="https://github.com/databricks/databricks-sdk-go/releases">github.com/databricks/databricks-sdk-go's
releases</a>.</em></p>
<blockquote>
<h2>v0.49.0</h2>
<h3>API Changes:</h3>
<ul>
<li>Added <a
href="https://pkg.go.dev/github.com/databricks/databricks-sdk-go/service/settings#DisableLegacyDbfsAPI">w.DisableLegacyDbfs</a>
workspace-level service.</li>
<li>Added <code>UnityCatalogProvisioningState</code> field for <a
href="https://pkg.go.dev/github.com/databricks/databricks-sdk-go/service/catalog#OnlineTable">catalog.OnlineTable</a>.</li>
<li>Added <code>IsTruncated</code> field for <a
href="https://pkg.go.dev/github.com/databricks/databricks-sdk-go/service/dashboards#Result">dashboards.Result</a>.</li>
<li>Added <code>EffectiveBudgetPolicyId</code> field for <a
href="https://pkg.go.dev/github.com/databricks/databricks-sdk-go/service/jobs#BaseJob">jobs.BaseJob</a>.</li>
<li>Added <code>BudgetPolicyId</code> field for <a
href="https://pkg.go.dev/github.com/databricks/databricks-sdk-go/service/jobs#CreateJob">jobs.CreateJob</a>.</li>
<li>Added <code>EffectiveBudgetPolicyId</code> field for <a
href="https://pkg.go.dev/github.com/databricks/databricks-sdk-go/service/jobs#Job">jobs.Job</a>.</li>
<li>Added <code>BudgetPolicyId</code> field for <a
href="https://pkg.go.dev/github.com/databricks/databricks-sdk-go/service/jobs#JobSettings">jobs.JobSettings</a>.</li>
<li>Added <code>BudgetPolicyId</code> field for <a
href="https://pkg.go.dev/github.com/databricks/databricks-sdk-go/service/jobs#SubmitRun">jobs.SubmitRun</a>.</li>
<li>Added <code>Report</code> field for <a
href="https://pkg.go.dev/github.com/databricks/databricks-sdk-go/service/pipelines#IngestionConfig">pipelines.IngestionConfig</a>.</li>
<li>Added <code>SequenceBy</code> field for <a
href="https://pkg.go.dev/github.com/databricks/databricks-sdk-go/service/pipelines#TableSpecificConfig">pipelines.TableSpecificConfig</a>.</li>
<li>Added <code>NotifyOnOk</code> field for <a
href="https://pkg.go.dev/github.com/databricks/databricks-sdk-go/service/sql#Alert">sql.Alert</a>.</li>
<li>Added <code>NotifyOnOk</code> field for <a
href="https://pkg.go.dev/github.com/databricks/databricks-sdk-go/service/sql#CreateAlertRequestAlert">sql.CreateAlertRequestAlert</a>.</li>
<li>Added <code>NotifyOnOk</code> field for <a
href="https://pkg.go.dev/github.com/databricks/databricks-sdk-go/service/sql#ListAlertsResponseAlert">sql.ListAlertsResponseAlert</a>.</li>
<li>Added <code>NotifyOnOk</code> field for <a
href="https://pkg.go.dev/github.com/databricks/databricks-sdk-go/service/sql#UpdateAlertRequestAlert">sql.UpdateAlertRequestAlert</a>.</li>
</ul>
<p>OpenAPI SHA: cf9c61453990df0f9453670f2fe68e1b128647a2, Date:
2024-10-14</p>
</blockquote>
</details>
<details>
<summary>Changelog</summary>
<p><em>Sourced from <a
href="https://github.com/databricks/databricks-sdk-go/blob/main/CHANGELOG.md">github.com/databricks/databricks-sdk-go's
changelog</a>.</em></p>
<blockquote>
<h2>[Release] Release v0.49.0</h2>
<h3>API Changes:</h3>
<ul>
<li>Added <a
href="https://pkg.go.dev/github.com/databricks/databricks-sdk-go/service/settings#DisableLegacyDbfsAPI">w.DisableLegacyDbfs</a>
workspace-level service.</li>
<li>Added <code>UnityCatalogProvisioningState</code> field for <a
href="https://pkg.go.dev/github.com/databricks/databricks-sdk-go/service/catalog#OnlineTable">catalog.OnlineTable</a>.</li>
<li>Added <code>IsTruncated</code> field for <a
href="https://pkg.go.dev/github.com/databricks/databricks-sdk-go/service/dashboards#Result">dashboards.Result</a>.</li>
<li>Added <code>EffectiveBudgetPolicyId</code> field for <a
href="https://pkg.go.dev/github.com/databricks/databricks-sdk-go/service/jobs#BaseJob">jobs.BaseJob</a>.</li>
<li>Added <code>BudgetPolicyId</code> field for <a
href="https://pkg.go.dev/github.com/databricks/databricks-sdk-go/service/jobs#CreateJob">jobs.CreateJob</a>.</li>
<li>Added <code>EffectiveBudgetPolicyId</code> field for <a
href="https://pkg.go.dev/github.com/databricks/databricks-sdk-go/service/jobs#Job">jobs.Job</a>.</li>
<li>Added <code>BudgetPolicyId</code> field for <a
href="https://pkg.go.dev/github.com/databricks/databricks-sdk-go/service/jobs#JobSettings">jobs.JobSettings</a>.</li>
<li>Added <code>BudgetPolicyId</code> field for <a
href="https://pkg.go.dev/github.com/databricks/databricks-sdk-go/service/jobs#SubmitRun">jobs.SubmitRun</a>.</li>
<li>Added <code>Report</code> field for <a
href="https://pkg.go.dev/github.com/databricks/databricks-sdk-go/service/pipelines#IngestionConfig">pipelines.IngestionConfig</a>.</li>
<li>Added <code>SequenceBy</code> field for <a
href="https://pkg.go.dev/github.com/databricks/databricks-sdk-go/service/pipelines#TableSpecificConfig">pipelines.TableSpecificConfig</a>.</li>
<li>Added <code>NotifyOnOk</code> field for <a
href="https://pkg.go.dev/github.com/databricks/databricks-sdk-go/service/sql#Alert">sql.Alert</a>.</li>
<li>Added <code>NotifyOnOk</code> field for <a
href="https://pkg.go.dev/github.com/databricks/databricks-sdk-go/service/sql#CreateAlertRequestAlert">sql.CreateAlertRequestAlert</a>.</li>
<li>Added <code>NotifyOnOk</code> field for <a
href="https://pkg.go.dev/github.com/databricks/databricks-sdk-go/service/sql#ListAlertsResponseAlert">sql.ListAlertsResponseAlert</a>.</li>
<li>Added <code>NotifyOnOk</code> field for <a
href="https://pkg.go.dev/github.com/databricks/databricks-sdk-go/service/sql#UpdateAlertRequestAlert">sql.UpdateAlertRequestAlert</a>.</li>
</ul>
<p>OpenAPI SHA: cf9c61453990df0f9453670f2fe68e1b128647a2, Date:
2024-10-14</p>
</blockquote>
</details>
<details>
<summary>Commits</summary>
<ul>
<li><a
href="b47ecac9ea"><code>b47ecac</code></a>
[Release] Release v0.49.0 (<a
href="https://redirect.github.com/databricks/databricks-sdk-go/issues/1062">#1062</a>)</li>
<li>See full diff in <a
href="https://github.com/databricks/databricks-sdk-go/compare/v0.48.0...v0.49.0">compare
view</a></li>
</ul>
</details>
<br />

<details>
<summary>Most Recent Ignore Conditions Applied to This Pull
Request</summary>

| Dependency Name | Ignore Conditions |
| --- | --- |
| github.com/databricks/databricks-sdk-go | [>= 0.28.a, < 0.29] |
</details>


[![Dependabot compatibility
score](https://dependabot-badges.githubapp.com/badges/compatibility_score?dependency-name=github.com/databricks/databricks-sdk-go&package-manager=go_modules&previous-version=0.48.0&new-version=0.49.0)](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores)

Dependabot will resolve any conflicts with this PR as long as you don't
alter it yourself. You can also trigger a rebase manually by commenting
`@dependabot rebase`.

[//]: # (dependabot-automerge-start)
[//]: # (dependabot-automerge-end)

---

<details>
<summary>Dependabot commands and options</summary>
<br />

You can trigger Dependabot actions by commenting on this PR:
- `@dependabot rebase` will rebase this PR
- `@dependabot recreate` will recreate this PR, overwriting any edits
that have been made to it
- `@dependabot merge` will merge this PR after your CI passes on it
- `@dependabot squash and merge` will squash and merge this PR after
your CI passes on it
- `@dependabot cancel merge` will cancel a previously requested merge
and block automerging
- `@dependabot reopen` will reopen this PR if it is closed
- `@dependabot close` will close this PR and stop Dependabot recreating
it. You can achieve the same result by closing it manually
- `@dependabot show <dependency name> ignore conditions` will show all
of the ignore conditions of the specified dependency
- `@dependabot ignore this major version` will close this PR and stop
Dependabot creating any more for this major version (unless you reopen
the PR or upgrade to it yourself)
- `@dependabot ignore this minor version` will close this PR and stop
Dependabot creating any more for this minor version (unless you reopen
the PR or upgrade to it yourself)
- `@dependabot ignore this dependency` will close this PR and stop
Dependabot creating any more for this dependency (unless you reopen the
PR or upgrade to it yourself)


</details>

---------

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Andrew Nester <andrew.nester@databricks.com>
 2024-10-22 09:37:01 +00:00
Andrew Nester ffdbec87cc
Added support for pip options in environment dependencies (#1842) 
## Changes
Added support for specifying pip options such as `--extra-index-url` and
etc. in environments dependencies

```
environments:
  - environment_key: Default
    spec:
      client: "1"
      dependencies:
        - --extra-index-url https://foo@bar.com/packages/smth somepackage
        - json==1.0.0
```

## Tests
Added regression test

---------

Co-authored-by: Pieter Noordhuis <pieter.noordhuis@databricks.com>
 2024-10-21 11:45:39 +00:00