bfe9cc4d50
create generated package
2024-09-04 11:56:52 +02:00
446463b9eb
update schema
2024-09-04 11:33:28 +02:00
9d9c6120a1
Merge remote-tracking branch 'origin' into improve/json-schema
2024-09-04 11:32:49 +02:00
ba3f004da1
schema local to generator
2024-09-04 11:32:39 +02:00
096123674a
Fix streaming of stdout, stdin, stderr in cobra test runner ( #1742 )
...
## Changes
We were not using the readers and writers set in the test fixtures in
the progress logger. This PR fixes that. It also modifies
`TestAccAbortBind`, which was implicitly relying on the bug.
I encountered this bug while working on
https://github.com/databricks/cli/pull/1672 .
## Tests
Manually.
From non-tty:
```
Error: failed to bind the resource, err: This bind operation requires user confirmation, but the current console does not support prompting. Please specify --auto-approve if you would like to skip prompts and proceed.
```
From tty, bind works as expected.
```
Confirm import changes? Changes will be remotely applied only after running 'bundle deploy'. [y/n]: y
Updating deployment state...
Successfully bound databricks_pipeline with an id '9d2dedbb-f522-4503-96ba-4bc4d5bfa77d'. Run 'bundle deploy' to deploy changes to your workspace
```
2024-09-02 13:43:17 +00:00
ed448815b4
PythonMutator: explain missing package error ( #1736 )
...
## Changes
Explain the error when the `databricks-pydabs` package is not installed
or the Python environment isn't correctly activated.
Example output:
```
Error: python mutator process failed: ".venv/bin/python3 -m databricks.bundles.build --phase load --input .../input.json --output .../output.json --diagnostics .../diagnostics.json: exit status 1", use --debug to enable logging
.../.venv/bin/python3: Error while finding module specification for 'databricks.bundles.build' (ModuleNotFoundError: No module named 'databricks')
Explanation: 'databricks-pydabs' library is not installed in the Python environment.
If using Python wheels, ensure that 'databricks-pydabs' is included in the dependencies,
and that the wheel is installed in the Python environment:
$ .venv/bin/pip install -e .
If using a virtual environment, ensure it is specified as the venv_path property in databricks.yml,
or activate the environment before running CLI commands:
experimental:
pydabs:
venv_path: .venv
```
## Tests
Unit tests
2024-09-02 09:49:30 +00:00
582558cac2
Do not suppress normalisation diagnostics for resolving variables ( #1740 )
...
## Changes
Tested on the following bundle configuration
```
bundle:
name: clusters
mode: development
variables:
webhook_notifications:
description: Webhook URL for notifications
type: complex
default:
on_failure:
id: 6a6c04c1-389c-4534-95af-b68b62a9dbe6
resources:
jobs:
test_job:
name: "Andrew Nester Test Job"
tasks:
- task_key: test_task
notebook_task:
notebook_path: "./src/test.py"
new_cluster:
num_workers: 2
node_type_id: "i3.xlarge"
autoscale:
min_workers: 2
max_workers: 7
spark_version: "12.2.x-scala2.12"
spark_conf:
"spark.executor.memory": "2g"
webhook_notifications: ${var.webhook_notifications}
```
bundle validate output is below
```
andrew.nester@HFW9Y94129 wheel % databricks bundle validate
Warning: expected sequence, found map
at resources.jobs.test_job.webhook_notifications.on_failure
in bundle.yml:11:9
Name: clusters
Target: default
Workspace:
User: andrew.nester@databricks.com
Path: /Users/andrew.nester@databricks.com/.bundle/clusters/default
```
**Note** that error correctly points to the variable
2024-09-02 09:17:18 +00:00
5d9910c8e0
Make lock optional in the JSON schema ( #1738 )
...
Fixes https://github.com/databricks/cli/issues/1561
2024-09-02 08:39:08 +00:00
70ce802518
PythonMutator: preserve normalize diagnostics ( #1735 )
...
## Changes
Preserve diagnostics if there are any errors or warnings when
PythonMutator normalizes output. If anything goes wrong during
conversion, diagnostics contain the relevant location and path.
## Tests
Unit tests
2024-08-30 13:29:00 +00:00
5fac7edcdf
Pass along $AZURE_CONFIG_FILE to Terraform process ( #1734 )
...
## Changes
This ensures that the CLI and Terraform can both use an Azure CLI
session configured under a non-standard path. This is the default
behavior on Azure DevOps when using the AzureCLI@2 task.
Fixes #1722 .
## Tests
Unit test.
2024-08-29 14:41:12 +00:00
43ace69bb9
Consider serverless clusters as compatible for Python wheel tasks ( #1733 )
...
## Changes
Consider serverless clusters as compatible for Python wheel tasks.
Fixes a `Python wheel tasks require compute with DBR 13.3+ to include
local libraries` warning shown for serverless clusters
2024-08-29 12:47:44 +00:00
85459c1963
Improve error handling for /Volumes paths in mode: development ( #1716 )
...
## Changes
* Provide a more helpful error when using an artifact_path based on
/Volumes
* Allow the use of short_names in /Volumes paths
## Example cases
Example of a valid /Volumes artifact_path:
* `artifact_path:
/Volumes/catalog/schema/${workspace.current_user.short_name}/libs`
Example of an invalid /Volumes path (when using `mode: development`):
* `artifact_path: /Volumes/catalog/schema/libs`
* Resulting error: `artifact_path should contain the current username or
${workspace.current_user.short_name} to ensure uniqueness when using
'mode: development'`
2024-08-28 12:14:19 +00:00
70363836d5
Correctly mark PyPI package name specs with multiple specifiers as remote libraries ( #1725 )
...
Correctly mark pypi package name specs with multiple specifiers as
remote libraries
Fixes this https://github.com/databricks/cli/issues/1728
2024-08-28 11:39:06 +00:00
cb8d6a9f60
self pass 1
2024-08-27 19:50:43 +02:00
4141f4ea34
-
2024-08-27 17:49:09 +02:00
40f4d35c4a
cleanup todos
2024-08-27 17:45:20 +02:00
ad7503a176
cleanup todos
2024-08-27 17:40:36 +02:00
f194a5b86a
skip adding patterns for string schemas
2024-08-27 17:36:12 +02:00
ac60163c19
move gen file
2024-08-27 13:35:56 +02:00
e7fd063e8a
-
2024-08-27 13:10:13 +02:00
fcdccb359a
cleanu unused code
2024-08-27 12:47:49 +02:00
d07192fe03
merge
2024-08-27 11:54:41 +02:00
2dea889621
delete the old schema code
2024-08-27 11:53:29 +02:00
056d203236
Bump github.com/databricks/databricks-sdk-go from 0.44.0 to 0.45.0 ( #1719 )
...
Bumps
[github.com/databricks/databricks-sdk-go](https://github.com/databricks/databricks-sdk-go )
from 0.44.0 to 0.45.0.
<details>
<summary>Release notes</summary>
<p><em>Sourced from <a
href="https://github.com/databricks/databricks-sdk-go/releases ">github.com/databricks/databricks-sdk-go's
releases</a>.</em></p>
<blockquote>
<h2>v0.45.0</h2>
<h2>0.45.0</h2>
<h3>Bug Fixes</h3>
<ul>
<li>Add INVALID_STATE to error code mapping (<a
href="https://redirect.github.com/databricks/databricks-sdk-go/pull/1014 ">#1014</a>).</li>
<li>Do not specify <code>--tenant</code> flag when fetching managed
identity access token from the CLI (<a
href="https://redirect.github.com/databricks/databricks-sdk-go/pull/1021 ">#1021</a>).</li>
</ul>
<h3>Internal Changes</h3>
<ul>
<li>Add terraform aliases to Entity (<a
href="https://redirect.github.com/databricks/databricks-sdk-go/pull/1017 ">#1017</a>).</li>
<li>Added Service.NamedIdMap (<a
href="https://redirect.github.com/databricks/databricks-sdk-go/pull/1016 ">#1016</a>).</li>
<li>Fix billing test for budget configuration update (<a
href="https://redirect.github.com/databricks/databricks-sdk-go/pull/1019 ">#1019</a>).</li>
</ul>
<h3>API Changes:</h3>
<ul>
<li>Added <a
href="https://pkg.go.dev/github.com/databricks/databricks-sdk-go/service/compute#PolicyComplianceForClustersAPI ">w.PolicyComplianceForClusters</a>
workspace-level service.</li>
<li>Added <a
href="https://pkg.go.dev/github.com/databricks/databricks-sdk-go/service/jobs#PolicyComplianceForJobsAPI ">w.PolicyComplianceForJobs</a>
workspace-level service.</li>
<li>Added <a
href="https://pkg.go.dev/github.com/databricks/databricks-sdk-go/service/catalog#ResourceQuotasAPI ">w.ResourceQuotas</a>
workspace-level service.</li>
<li>Added <a
href="https://pkg.go.dev/github.com/databricks/databricks-sdk-go/service/catalog#GetQuotaRequest ">catalog.GetQuotaRequest</a>,
<a
href="https://pkg.go.dev/github.com/databricks/databricks-sdk-go/service/catalog#GetQuotaResponse ">catalog.GetQuotaResponse</a>,
<a
href="https://pkg.go.dev/github.com/databricks/databricks-sdk-go/service/catalog#ListQuotasRequest ">catalog.ListQuotasRequest</a>,
<a
href="https://pkg.go.dev/github.com/databricks/databricks-sdk-go/service/catalog#ListQuotasResponse ">catalog.ListQuotasResponse</a>
and <a
href="https://pkg.go.dev/github.com/databricks/databricks-sdk-go/service/catalog#QuotaInfo ">catalog.QuotaInfo</a>.</li>
<li>Added <a
href="https://pkg.go.dev/github.com/databricks/databricks-sdk-go/service/compute#ClusterCompliance ">compute.ClusterCompliance</a>,
<a
href="https://pkg.go.dev/github.com/databricks/databricks-sdk-go/service/compute#ClusterSettingsChange ">compute.ClusterSettingsChange</a>,
<a
href="https://pkg.go.dev/github.com/databricks/databricks-sdk-go/service/compute#EnforceClusterComplianceRequest ">compute.EnforceClusterComplianceRequest</a>,
<a
href="https://pkg.go.dev/github.com/databricks/databricks-sdk-go/service/compute#EnforceClusterComplianceResponse ">compute.EnforceClusterComplianceResponse</a>,
<a
href="https://pkg.go.dev/github.com/databricks/databricks-sdk-go/service/compute#GetClusterComplianceRequest ">compute.GetClusterComplianceRequest</a>,
<a
href="https://pkg.go.dev/github.com/databricks/databricks-sdk-go/service/compute#GetClusterComplianceResponse ">compute.GetClusterComplianceResponse</a>,
<a
href="https://pkg.go.dev/github.com/databricks/databricks-sdk-go/service/compute#ListClusterCompliancesRequest ">compute.ListClusterCompliancesRequest</a>
and <a
href="https://pkg.go.dev/github.com/databricks/databricks-sdk-go/service/compute#ListClusterCompliancesResponse ">compute.ListClusterCompliancesResponse</a>.</li>
<li>Added <a
href="https://pkg.go.dev/github.com/databricks/databricks-sdk-go/service/jobs#EnforcePolicyComplianceForJobResponseJobClusterSettingsChange ">jobs.EnforcePolicyComplianceForJobResponseJobClusterSettingsChange</a>,
<a
href="https://pkg.go.dev/github.com/databricks/databricks-sdk-go/service/jobs#EnforcePolicyComplianceRequest ">jobs.EnforcePolicyComplianceRequest</a>,
<a
href="https://pkg.go.dev/github.com/databricks/databricks-sdk-go/service/jobs#EnforcePolicyComplianceResponse ">jobs.EnforcePolicyComplianceResponse</a>,
<a
href="https://pkg.go.dev/github.com/databricks/databricks-sdk-go/service/jobs#GetPolicyComplianceRequest ">jobs.GetPolicyComplianceRequest</a>,
<a
href="https://pkg.go.dev/github.com/databricks/databricks-sdk-go/service/jobs#GetPolicyComplianceResponse ">jobs.GetPolicyComplianceResponse</a>,
<a
href="https://pkg.go.dev/github.com/databricks/databricks-sdk-go/service/jobs#JobCompliance ">jobs.JobCompliance</a>,
<a
href="https://pkg.go.dev/github.com/databricks/databricks-sdk-go/service/jobs#ListJobComplianceForPolicyResponse ">jobs.ListJobComplianceForPolicyResponse</a>
and <a
href="https://pkg.go.dev/github.com/databricks/databricks-sdk-go/service/jobs#ListJobComplianceRequest ">jobs.ListJobComplianceRequest</a>.</li>
<li>Added <code>Fallback</code> field for <a
href="https://pkg.go.dev/github.com/databricks/databricks-sdk-go/service/catalog#CreateExternalLocation ">catalog.CreateExternalLocation</a>.</li>
<li>Added <code>Fallback</code> field for <a
href="https://pkg.go.dev/github.com/databricks/databricks-sdk-go/service/catalog#ExternalLocationInfo ">catalog.ExternalLocationInfo</a>.</li>
<li>Added <code>Fallback</code> field for <a
href="https://pkg.go.dev/github.com/databricks/databricks-sdk-go/service/catalog#UpdateExternalLocation ">catalog.UpdateExternalLocation</a>.</li>
<li>Added <code>JobRunId</code> field for <a
href="https://pkg.go.dev/github.com/databricks/databricks-sdk-go/service/jobs#BaseRun ">jobs.BaseRun</a>.</li>
<li>Added <code>JobRunId</code> field for <a
href="https://pkg.go.dev/github.com/databricks/databricks-sdk-go/service/jobs#Run ">jobs.Run</a>.</li>
<li>Added <code>IncludeMetrics</code> field for <a
href="https://pkg.go.dev/github.com/databricks/databricks-sdk-go/service/sql#ListQueryHistoryRequest ">sql.ListQueryHistoryRequest</a>.</li>
<li>Added <code>StatementIds</code> field for <a
href="https://pkg.go.dev/github.com/databricks/databricks-sdk-go/service/sql#QueryFilter ">sql.QueryFilter</a>.</li>
<li>Removed <a
href="https://pkg.go.dev/github.com/databricks/databricks-sdk-go/service/sql#ContextFilter ">sql.ContextFilter</a>.</li>
<li>Removed <code>ContextFilter</code> field for <a
href="https://pkg.go.dev/github.com/databricks/databricks-sdk-go/service/sql#QueryFilter ">sql.QueryFilter</a>.</li>
<li>Removed <code>PipelineId</code> and <code>PipelineUpdateId</code>
fields for <a
href="https://pkg.go.dev/github.com/databricks/databricks-sdk-go/service/sql#QuerySource ">sql.QuerySource</a>.</li>
</ul>
<p>OpenAPI SHA: 3eae49b444cac5a0118a3503e5b7ecef7f96527a, Date:
2024-08-21</p>
</blockquote>
</details>
<details>
<summary>Changelog</summary>
<p><em>Sourced from <a
href="https://github.com/databricks/databricks-sdk-go/blob/main/CHANGELOG.md ">github.com/databricks/databricks-sdk-go's
changelog</a>.</em></p>
<blockquote>
<h2>[Release] Release v0.45.0</h2>
<h3>Bug Fixes</h3>
<ul>
<li>Add INVALID_STATE to error code mapping (<a
href="https://redirect.github.com/databricks/databricks-sdk-go/pull/1014 ">#1014</a>).</li>
<li>Do not specify <code>--tenant</code> flag when fetching managed
identity access token from the CLI (<a
href="https://redirect.github.com/databricks/databricks-sdk-go/pull/1021 ">#1021</a>).</li>
</ul>
<h3>Internal Changes</h3>
<ul>
<li>Add terraform aliases to Entity (<a
href="https://redirect.github.com/databricks/databricks-sdk-go/pull/1017 ">#1017</a>).</li>
<li>Added Service.NamedIdMap (<a
href="https://redirect.github.com/databricks/databricks-sdk-go/pull/1016 ">#1016</a>).</li>
<li>Fix billing test for budget configuration update (<a
href="https://redirect.github.com/databricks/databricks-sdk-go/pull/1019 ">#1019</a>).</li>
</ul>
<h3>API Changes:</h3>
<ul>
<li>Added <a
href="https://pkg.go.dev/github.com/databricks/databricks-sdk-go/service/compute#PolicyComplianceForClustersAPI ">w.PolicyComplianceForClusters</a>
workspace-level service.</li>
<li>Added <a
href="https://pkg.go.dev/github.com/databricks/databricks-sdk-go/service/jobs#PolicyComplianceForJobsAPI ">w.PolicyComplianceForJobs</a>
workspace-level service.</li>
<li>Added <a
href="https://pkg.go.dev/github.com/databricks/databricks-sdk-go/service/catalog#ResourceQuotasAPI ">w.ResourceQuotas</a>
workspace-level service.</li>
<li>Added <a
href="https://pkg.go.dev/github.com/databricks/databricks-sdk-go/service/catalog#GetQuotaRequest ">catalog.GetQuotaRequest</a>,
<a
href="https://pkg.go.dev/github.com/databricks/databricks-sdk-go/service/catalog#GetQuotaResponse ">catalog.GetQuotaResponse</a>,
<a
href="https://pkg.go.dev/github.com/databricks/databricks-sdk-go/service/catalog#ListQuotasRequest ">catalog.ListQuotasRequest</a>,
<a
href="https://pkg.go.dev/github.com/databricks/databricks-sdk-go/service/catalog#ListQuotasResponse ">catalog.ListQuotasResponse</a>
and <a
href="https://pkg.go.dev/github.com/databricks/databricks-sdk-go/service/catalog#QuotaInfo ">catalog.QuotaInfo</a>.</li>
<li>Added <a
href="https://pkg.go.dev/github.com/databricks/databricks-sdk-go/service/compute#ClusterCompliance ">compute.ClusterCompliance</a>,
<a
href="https://pkg.go.dev/github.com/databricks/databricks-sdk-go/service/compute#ClusterSettingsChange ">compute.ClusterSettingsChange</a>,
<a
href="https://pkg.go.dev/github.com/databricks/databricks-sdk-go/service/compute#EnforceClusterComplianceRequest ">compute.EnforceClusterComplianceRequest</a>,
<a
href="https://pkg.go.dev/github.com/databricks/databricks-sdk-go/service/compute#EnforceClusterComplianceResponse ">compute.EnforceClusterComplianceResponse</a>,
<a
href="https://pkg.go.dev/github.com/databricks/databricks-sdk-go/service/compute#GetClusterComplianceRequest ">compute.GetClusterComplianceRequest</a>,
<a
href="https://pkg.go.dev/github.com/databricks/databricks-sdk-go/service/compute#GetClusterComplianceResponse ">compute.GetClusterComplianceResponse</a>,
<a
href="https://pkg.go.dev/github.com/databricks/databricks-sdk-go/service/compute#ListClusterCompliancesRequest ">compute.ListClusterCompliancesRequest</a>
and <a
href="https://pkg.go.dev/github.com/databricks/databricks-sdk-go/service/compute#ListClusterCompliancesResponse ">compute.ListClusterCompliancesResponse</a>.</li>
<li>Added <a
href="https://pkg.go.dev/github.com/databricks/databricks-sdk-go/service/jobs#EnforcePolicyComplianceForJobResponseJobClusterSettingsChange ">jobs.EnforcePolicyComplianceForJobResponseJobClusterSettingsChange</a>,
<a
href="https://pkg.go.dev/github.com/databricks/databricks-sdk-go/service/jobs#EnforcePolicyComplianceRequest ">jobs.EnforcePolicyComplianceRequest</a>,
<a
href="https://pkg.go.dev/github.com/databricks/databricks-sdk-go/service/jobs#EnforcePolicyComplianceResponse ">jobs.EnforcePolicyComplianceResponse</a>,
<a
href="https://pkg.go.dev/github.com/databricks/databricks-sdk-go/service/jobs#GetPolicyComplianceRequest ">jobs.GetPolicyComplianceRequest</a>,
<a
href="https://pkg.go.dev/github.com/databricks/databricks-sdk-go/service/jobs#GetPolicyComplianceResponse ">jobs.GetPolicyComplianceResponse</a>,
<a
href="https://pkg.go.dev/github.com/databricks/databricks-sdk-go/service/jobs#JobCompliance ">jobs.JobCompliance</a>,
<a
href="https://pkg.go.dev/github.com/databricks/databricks-sdk-go/service/jobs#ListJobComplianceForPolicyResponse ">jobs.ListJobComplianceForPolicyResponse</a>
and <a
href="https://pkg.go.dev/github.com/databricks/databricks-sdk-go/service/jobs#ListJobComplianceRequest ">jobs.ListJobComplianceRequest</a>.</li>
<li>Added <code>Fallback</code> field for <a
href="https://pkg.go.dev/github.com/databricks/databricks-sdk-go/service/catalog#CreateExternalLocation ">catalog.CreateExternalLocation</a>.</li>
<li>Added <code>Fallback</code> field for <a
href="https://pkg.go.dev/github.com/databricks/databricks-sdk-go/service/catalog#ExternalLocationInfo ">catalog.ExternalLocationInfo</a>.</li>
<li>Added <code>Fallback</code> field for <a
href="https://pkg.go.dev/github.com/databricks/databricks-sdk-go/service/catalog#UpdateExternalLocation ">catalog.UpdateExternalLocation</a>.</li>
<li>Added <code>JobRunId</code> field for <a
href="https://pkg.go.dev/github.com/databricks/databricks-sdk-go/service/jobs#BaseRun ">jobs.BaseRun</a>.</li>
<li>Added <code>JobRunId</code> field for <a
href="https://pkg.go.dev/github.com/databricks/databricks-sdk-go/service/jobs#Run ">jobs.Run</a>.</li>
<li>Added <code>IncludeMetrics</code> field for <a
href="https://pkg.go.dev/github.com/databricks/databricks-sdk-go/service/sql#ListQueryHistoryRequest ">sql.ListQueryHistoryRequest</a>.</li>
<li>Added <code>StatementIds</code> field for <a
href="https://pkg.go.dev/github.com/databricks/databricks-sdk-go/service/sql#QueryFilter ">sql.QueryFilter</a>.</li>
<li>Removed <a
href="https://pkg.go.dev/github.com/databricks/databricks-sdk-go/service/sql#ContextFilter ">sql.ContextFilter</a>.</li>
<li>Removed <code>ContextFilter</code> field for <a
href="https://pkg.go.dev/github.com/databricks/databricks-sdk-go/service/sql#QueryFilter ">sql.QueryFilter</a>.</li>
<li>Removed <code>PipelineId</code> and <code>PipelineUpdateId</code>
fields for <a
href="https://pkg.go.dev/github.com/databricks/databricks-sdk-go/service/sql#QuerySource ">sql.QuerySource</a>.</li>
</ul>
<p>OpenAPI SHA: 3eae49b444cac5a0118a3503e5b7ecef7f96527a, Date:
2024-08-21</p>
</blockquote>
</details>
<details>
<summary>Commits</summary>
<ul>
<li><a
href="6d867882d0https://redirect.github.com/databricks/databricks-sdk-go/issues/1023 ">#1023</a>)</li>
<li><a
href="ba4489b946f6248097d1https://redirect.github.com/databricks/databricks-sdk-go/issues/1019 ">#1019</a>)</li>
<li><a
href="27a5055609https://redirect.github.com/databricks/databricks-sdk-go/issues/1017 ">#1017</a>)</li>
<li><a
href="382a38d380https://redirect.github.com/databricks/databricks-sdk-go/issues/1016 ">#1016</a>)</li>
<li><a
href="1ef9931dc9https://redirect.github.com/databricks/databricks-sdk-go/issues/1014 ">#1014</a>)</li>
<li>See full diff in <a
href="https://github.com/databricks/databricks-sdk-go/compare/v0.44.0...v0.45.0 ">compare
view</a></li>
</ul>
</details>
<br />
<details>
<summary>Most Recent Ignore Conditions Applied to This Pull
Request</summary>
| Dependency Name | Ignore Conditions |
| --- | --- |
| github.com/databricks/databricks-sdk-go | [>= 0.28.a, < 0.29] |
</details>
[](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores )
Dependabot will resolve any conflicts with this PR as long as you don't
alter it yourself. You can also trigger a rebase manually by commenting
`@dependabot rebase`.
[//]: # (dependabot-automerge-start)
[//]: # (dependabot-automerge-end)
---
<details>
<summary>Dependabot commands and options</summary>
<br />
You can trigger Dependabot actions by commenting on this PR:
- `@dependabot rebase` will rebase this PR
- `@dependabot recreate` will recreate this PR, overwriting any edits
that have been made to it
- `@dependabot merge` will merge this PR after your CI passes on it
- `@dependabot squash and merge` will squash and merge this PR after
your CI passes on it
- `@dependabot cancel merge` will cancel a previously requested merge
and block automerging
- `@dependabot reopen` will reopen this PR if it is closed
- `@dependabot close` will close this PR and stop Dependabot recreating
it. You can achieve the same result by closing it manually
- `@dependabot show <dependency name> ignore conditions` will show all
of the ignore conditions of the specified dependency
- `@dependabot ignore this major version` will close this PR and stop
Dependabot creating any more for this major version (unless you reopen
the PR or upgrade to it yourself)
- `@dependabot ignore this minor version` will close this PR and stop
Dependabot creating any more for this minor version (unless you reopen
the PR or upgrade to it yourself)
- `@dependabot ignore this dependency` will close this PR and stop
Dependabot creating any more for this dependency (unless you reopen the
PR or upgrade to it yourself)
</details>
---------
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Andrew Nester <andrew.nester@databricks.com>
2024-08-27 08:54:05 +00:00
11dfdc036d
more iteration
2024-08-26 20:16:45 +02:00
783e05c939
Do not treat empty path as a local path ( #1717 )
...
## Changes
Fixes issue introduced here https://github.com/databricks/cli/pull/1699
where PyPi packages were treated as local library.
The reason is that `libraryPath` returns an empty string as a path for
PyPi packages and then `IsLibraryLocal` treated empty string as local
path.
Both of these functions are fixed in this PR.
## Tests
Added regression test
2024-08-26 10:03:56 +00:00
84b47745e4
Ignore CLI version check on development builds of the CLI ( #1714 )
...
## Changes
This changes makes sure we ignore CLI version check on development
builds of the CLI.
Before:
```
$ cat databricks.yml | grep cli_version
databricks_cli_version: ">= 0.223.1"
$ cli bundle deploy
Error: Databricks CLI version constraint not satisfied. Required: >= 0.223.1, current: 0.0.0-dev+06b169284737
```
after
```
...
$ cli bundle deploy
...
Warning: Ignoring Databricks CLI version constraint for development build. Required: >= 0.223.1, current: 0.0.0-dev+d52d6f08fcd5
```
## Tests
<!-- How is this tested? -->
2024-08-23 10:13:21 +00:00
790731f981
remove generic schema test
2024-08-22 17:42:19 +02:00
39efb705de
-
2024-08-22 17:40:43 +02:00
7c70179091
add self referential tesT
2024-08-22 17:38:24 +02:00
08133126ae
Merge remote-tracking branch 'origin' into improve/json-schema
2024-08-22 17:27:49 +02:00
6e8cd835a3
Add paths field to bundle sync configuration ( #1694 )
...
## Changes
This field allows a user to configure paths to synchronize to the
workspace.
Allowed values are relative paths to files and directories anchored at
the directory where the field is set. If one or more values traverse up
the directory tree (to an ancestor of the bundle root directory), the
CLI will dynamically determine the root path to use to ensure that the
file tree structure remains intact.
For example, given a `databricks.yml` in `my_bundle` that includes:
```yaml
sync:
paths:
- ../common
- .
```
Then upon synchronization, the workspace will look like:
```
.
├── common
│ └── lib.py
└── my_bundle
├── databricks.yml
└── notebook.py
```
If not set behavior remains identical.
## Tests
* Newly added unit tests for the mutators and under `bundle/tests`.
* Manually confirmed a bundle without this configuration works the same.
* Manually confirmed a bundle with this configuration works.
2024-08-21 15:33:25 +00:00
f5df211320
Fix prefix preset used for UC schemas ( #1704 )
...
## Changes
In https://github.com/databricks/cli/pull/1490 we regressed and started
using the development mode prefix for UC schemas regardless of the mode
of the bundle target.
This PR fixes the regression and adds a regression test
## Tests
Failing integration tests pass now.
2024-08-21 12:53:54 +00:00
192f33bb13
[DAB] Add support for requirements libraries in Job Tasks ( #1543 )
...
## Changes
While experimenting with DAB I discovered that requirements libraries
are being ignored.
One thing worth mentioning is that `bundle validate` runs successfully,
but `bundle deploy` fails. This PR only covers the second part.
## Tests
<!-- How is this tested? -->
Added a unit test
2024-08-21 10:03:56 +00:00
c775d251ed
Improves detection of PyPI package names in environment dependencies ( #1699 )
...
## Changes
Improves detection of PyPi package names in environment dependencies
## Tests
Added unit tests
2024-08-21 08:22:35 +00:00
460eeb928d
more tests
2024-08-20 21:11:42 +02:00
b023ba0dd4
removed more tests
2024-08-20 19:57:57 +02:00
b89928513e
-
2024-08-20 19:10:50 +02:00
7db32fc211
-
2024-08-20 19:09:21 +02:00
65f1c750f6
-
2024-08-20 18:46:23 +02:00
6d2f88282f
more progress on the internal functionality
2024-08-20 18:40:30 +02:00
43325fdd0a
add some more todos:
2024-08-20 15:39:31 +02:00
75f252e51c
Improve JSON schema
2024-08-20 15:34:49 +02:00
44902fa350
Make `pydabs/venv_path` optional ( #1687 )
...
## Changes
Make `pydabs/venv_path` optional. When not specified, CLI detects the
Python interpreter using `python.DetectExecutable`, the same way as for
`artifacts`. `python.DetectExecutable` works correctly if a virtual
environment is activated or `python3` is available on PATH through other
means.
Extract the venv detection code from PyDABs into `libs/python/detect`.
This code will be used when we implement the `python/venv_path` section
in `databricks.yml`.
## Tests
Unit tests and manually
---------
Co-authored-by: Pieter Noordhuis <pcnoordhuis@gmail.com>
2024-08-20 13:26:57 +00:00
af5048e73e
Share test initializer in common helper function ( #1695 )
...
## Changes
These tests inadvertently re-ran mutators, the first time through
`loadTarget` and the second time by running `phases.Initialize()`
themselves. Some of the mutators that are executed in
`phases.Initialize()` are also run as part of `loadTarget`. This is
overdue a refactor to make it unambiguous what runs when. Until then,
this removes the duplicated execution.
## Tests
Unit tests pass.
2024-08-20 12:54:56 +00:00
6771ba09a6
Correctly mark package names with versions as remote libraries ( #1697 )
...
## Changes
Fixes https://github.com/databricks/setup-cli/issues/124
## Tests
Added regression test
2024-08-20 09:33:03 +00:00
242d4b51ed
Report all empty resources present in error diagnostic ( #1685 )
...
## Changes
This PR addressed post-merge feedback from
https://github.com/databricks/cli/pull/1673 .
## Tests
Unit tests, and manually.
```
Error: experiment undefined-experiment is not defined
at resources.experiments.undefined-experiment
in databricks.yml:11:26
Error: job undefined-job is not defined
at resources.jobs.undefined-job
in databricks.yml:6:19
Error: pipeline undefined-pipeline is not defined
at resources.pipelines.undefined-pipeline
in databricks.yml:14:24
Name: undefined-job
Target: default
Found 3 errors
```
2024-08-20 00:22:00 +00:00
78d0ac5c6a
Add configurable presets for name prefixes, tags, etc. ( #1490 )
...
## Changes
This adds configurable transformations based on the transformations
currently seen in `mode: development`.
Example databricks.yml showcasing how some transformations:
```
bundle:
name: my_bundle
targets:
dev:
presets:
prefix: "myprefix_" # prefix all resource names with myprefix_
pipelines_development: true # set development to true by default for pipelines
trigger_pause_status: PAUSED # set pause_status to PAUSED by default for all triggers and schedules
jobs_max_concurrent_runs: 10 # set max_concurrent runs to 10 by default for all jobs
tags:
dev: true
```
## Tests
* Existing process_target_mode tests that were adapted to use this new
code
* Unit tests specific for the new mutator
* Unit tests for config loading and merging
* Manual e2e testing
2024-08-19 18:18:50 +00:00
07627023f5
Pause continuous pipelines when 'mode: development' is used ( #1590 )
...
## Changes
This makes it so that the pipelines `continuous` property is set to
false by default when using `mode: development`.
2024-08-19 16:27:57 +00:00
2b8cbc31cf
Pass through paths argument to libs/sync ( #1689 )
...
## Changes
Requires #1684 .
## Tests
Ran the sync integration tests.
2024-08-19 15:41:02 +00:00
Shreyas Goenka
shreyas-goenka
Gleb Kanterov
Andrew Nester
Pieter Noordhuis
Lennart Kats (databricks)
dependabot[bot]
Witold Czaplewski