b323703c1b
## Changes This PR adds a warning validating that the configuration for a single node cluster is valid for interactive, job, job-task, and pipeline clusters. Note: We skip the validation if a cluster policy is configured because the policy is likely to configure `spark_conf` / `custom_tags` itself. Note: Terrform originally only had validation for interactive, job, and job-task clusters. This PR adding the validation for pipeline clusters as well is new. This PR follows the same logic as we used to have in Terraform. The validation was removed from Terraform because we had no way to demote the error to a warning: https://github.com/databricks/terraform-provider-databricks/pull/4222 ### Background Single-node clusters require `spark_conf` and `custom_tags` to be correctly set in the cluster definition for them to function optimally. The cluster will be created even if incorrectly configured, but its performance will not be great. For example, if both `spark_conf` and `custom_tags` are not set and `num_workers` is 0, then only the driver process will be launched on the cluster compute instance thus leading to sub-optimal utilization of available compute resources and no parallelization across worker processes when processing a spark query. ### Issue This PR addresses some issues reported in https://github.com/databricks/cli/issues/1546 ## Tests Unit tests and manually. Example output of the warning: ``` ➜ bundle-playground git:(master) ✗ cli bundle validate Warning: Single node cluster is not correctly configured at resources.pipelines.bar.clusters[0] in databricks.yml:29:11 num_workers should be 0 only for single-node clusters. To create a valid single node cluster please ensure that the following properties are correctly set in the cluster specification: spark_conf: spark.databricks.cluster.profile: singleNode spark.master: local[*] custom_tags: ResourceClass: SingleNode Name: foobar Target: default Workspace: User: shreyas.goenka@databricks.com Path: /Workspace/Users/shreyas.goenka@databricks.com/.bundle/foobar/default Found 1 warning ``` |
||
|---|---|---|
| .codegen | ||
| .github | ||
| .vscode | ||
| bundle | ||
| cmd | ||
| docker | ||
| docs | ||
| internal | ||
| libs | ||
| .codegen.json | ||
| .gitattributes | ||
| .gitignore | ||
| .goreleaser.yaml | ||
| .mockery.yaml | ||
| CHANGELOG.md | ||
| Dockerfile | ||
| LICENSE | ||
| Makefile | ||
| NOTICE | ||
| README.md | ||
| go.mod | ||
| go.sum | ||
| main.go | ||
| main_test.go | ||
README.md
Databricks CLI
This project is in Public Preview.
Documentation is available at https://docs.databricks.com/dev-tools/cli/databricks-cli.html.
Installation
This CLI is packaged as a dependency-free binary executable and may be located in any directory. See https://github.com/databricks/cli/releases for releases and the Databricks documentation for detailed information about installing the CLI.
Homebrew
We maintain a Homebrew tap for installing the Databricks CLI. You can find instructions for how to install, upgrade and downgrade the CLI using Homebrew here.
Docker
You can use the CLI via a Docker image by pulling the image from ghcr.io. You can find all available versions
at: https://github.com/databricks/cli/pkgs/container/cli.
docker pull ghcr.io/databricks/cli:latest
Example of how to run the CLI using the Docker image. More documentation is available at https://docs.databricks.com/dev-tools/bundles/airgapped-environment.html.
docker run -e DATABRICKS_HOST=$YOUR_HOST_URL -e DATABRICKS_TOKEN=$YOUR_TOKEN ghcr.io/databricks/cli:latest current-user me
Authentication
This CLI follows the Databricks Unified Authentication principles.
You can find a detailed description at https://github.com/databricks/databricks-sdk-go#authentication.
Privacy Notice
Databricks CLI use is subject to the Databricks License and Databricks Privacy Notice, including any Usage Data provisions.