mirror of https://github.com/databricks/cli.git
2847533e1e
## Changes
This PR adds support for UC volumes to DABs.
### Can I use a UC volume managed by DABs in `artifact_path`?
Yes, but we require the volume to exist before being referenced in
`artifact_path`. Otherwise you'll see an error that the volume does not
exist. For this case, this PR also adds a warning if we detect that the
UC volume is defined in the DAB itself, which informs the user to deploy
the UC volume in a separate deployment first before using it in
`artifact_path`.
We cannot create the UC volume and then upload the artifacts to it in
the same `bundle deploy` because `bundle deploy` always uploads the
artifacts to `artifact_path` before materializing any resources defined
in the bundle. Supporting this in a single deployment requires us to
migrate away from our dependency on the Databricks Terraform provider to
manage the CRUD lifecycle of DABs resources.
### Why do we not support `preset.name_prefix` for UC volumes?
UC volumes will not have a `dev_shreyas_goenka` prefix added in `mode:
development`. Configuring `presets.name_prefix` will be a no-op for UC
volumes. We have decided not to support prefixing for UC resources. This
is because:
1. UC provides its own namespace hierarchy that is independent of DABs.
2. Users can always manually use `${workspace.current_user.short_name}`
to configure the prefixes manually.
Customers often manually set up a UC hierarchy for dev and prod,
including a schema or catalog per developer. Thus, it's often
unnecessary for us to add prefixing in `mode: development` by default
for UC resources.
In retrospect, supporting prefixing for UC schemas and registered models
was a mistake and will be removed in a future release of DABs.
## Tests
Unit, integration test, and manually.
### Manual Testing cases:
1. UC volume does not exist:
```
➜ bundle-playground git:(master) ✗ cli bundle deploy
Error: failed to fetch metadata for the UC volume /Volumes/main/caps/my_volume that is configured in the artifact_path: Not Found
```
2. UC Volume does not exist, but is defined in the DAB
```
➜ bundle-playground git:(master) ✗ cli bundle deploy
Error: failed to fetch metadata for the UC volume /Volumes/main/caps/managed_by_dab that is configured in the artifact_path: Not Found
Warning: You might be using a UC volume in your artifact_path that is managed by this bundle but which has not been deployed yet. Please deploy the UC volume in a separate bundle deploy before using it in the artifact_path.
at resources.volumes.bar
in databricks.yml:24:7
```
---------
Co-authored-by: Pieter Noordhuis <pieter.noordhuis@databricks.com>
|
||
|---|---|---|
| .. | ||
| paths | ||
| python | ||
| apply_presets.go | ||
| apply_presets_test.go | ||
| compute_id_compat.go | ||
| compute_id_compate_test.go | ||
| configure_dashboard_defaults.go | ||
| configure_dashboard_defaults_test.go | ||
| configure_wsfs.go | ||
| configure_wsfs_test.go | ||
| default_queueing.go | ||
| default_queueing_test.go | ||
| default_target.go | ||
| default_target_test.go | ||
| default_workspace_paths.go | ||
| default_workspace_paths_test.go | ||
| default_workspace_root.go | ||
| default_workspace_root_test.go | ||
| environments_compat.go | ||
| environments_compat_test.go | ||
| expand_pipeline_glob_paths.go | ||
| expand_pipeline_glob_paths_test.go | ||
| expand_workspace_root.go | ||
| expand_workspace_root_test.go | ||
| initialize_urls.go | ||
| initialize_urls_test.go | ||
| initialize_variables.go | ||
| initialize_variables_test.go | ||
| initialize_workspace_client.go | ||
| load_git_details.go | ||
| merge_job_clusters.go | ||
| merge_job_clusters_test.go | ||
| merge_job_parameters.go | ||
| merge_job_parameters_test.go | ||
| merge_job_tasks.go | ||
| merge_job_tasks_test.go | ||
| merge_pipeline_clusters.go | ||
| merge_pipeline_clusters_test.go | ||
| mutator.go | ||
| noop.go | ||
| override_compute.go | ||
| override_compute_test.go | ||
| populate_current_user.go | ||
| populate_current_user_test.go | ||
| prepend_workspace_prefix.go | ||
| prepend_workspace_prefix_test.go | ||
| process_target_mode.go | ||
| process_target_mode_test.go | ||
| resolve_resource_references.go | ||
| resolve_resource_references_test.go | ||
| resolve_variable_references.go | ||
| resolve_variable_references_test.go | ||
| rewrite_sync_paths.go | ||
| rewrite_sync_paths_test.go | ||
| rewrite_workspace_prefix.go | ||
| rewrite_workspace_prefix_test.go | ||
| run_as.go | ||
| run_as_test.go | ||
| select_default_target.go | ||
| select_default_target_test.go | ||
| select_target.go | ||
| select_target_test.go | ||
| set_variables.go | ||
| set_variables_test.go | ||
| sync_default_path.go | ||
| sync_default_path_test.go | ||
| sync_infer_root.go | ||
| sync_infer_root_internal_test.go | ||
| sync_infer_root_test.go | ||
| translate_paths.go | ||
| translate_paths_artifacts.go | ||
| translate_paths_dashboards.go | ||
| translate_paths_dashboards_test.go | ||
| translate_paths_jobs.go | ||
| translate_paths_pipelines.go | ||
| translate_paths_test.go | ||
| validate_git_details.go | ||
| validate_git_details_test.go | ||
| verify_cli_version.go | ||
| verify_cli_version_test.go | ||