mirror of https://github.com/databricks/cli.git
b323703c1b
## Changes This PR adds a warning validating that the configuration for a single node cluster is valid for interactive, job, job-task, and pipeline clusters. Note: We skip the validation if a cluster policy is configured because the policy is likely to configure `spark_conf` / `custom_tags` itself. Note: Terrform originally only had validation for interactive, job, and job-task clusters. This PR adding the validation for pipeline clusters as well is new. This PR follows the same logic as we used to have in Terraform. The validation was removed from Terraform because we had no way to demote the error to a warning: https://github.com/databricks/terraform-provider-databricks/pull/4222 ### Background Single-node clusters require `spark_conf` and `custom_tags` to be correctly set in the cluster definition for them to function optimally. The cluster will be created even if incorrectly configured, but its performance will not be great. For example, if both `spark_conf` and `custom_tags` are not set and `num_workers` is 0, then only the driver process will be launched on the cluster compute instance thus leading to sub-optimal utilization of available compute resources and no parallelization across worker processes when processing a spark query. ### Issue This PR addresses some issues reported in https://github.com/databricks/cli/issues/1546 ## Tests Unit tests and manually. Example output of the warning: ``` ➜ bundle-playground git:(master) ✗ cli bundle validate Warning: Single node cluster is not correctly configured at resources.pipelines.bar.clusters[0] in databricks.yml:29:11 num_workers should be 0 only for single-node clusters. To create a valid single node cluster please ensure that the following properties are correctly set in the cluster specification: spark_conf: spark.databricks.cluster.profile: singleNode spark.master: local[*] custom_tags: ResourceClass: SingleNode Name: foobar Target: default Workspace: User: shreyas.goenka@databricks.com Path: /Workspace/Users/shreyas.goenka@databricks.com/.bundle/foobar/default Found 1 warning ``` |
||
|---|---|---|
| .. | ||
| artifacts | ||
| config | ||
| deploy | ||
| env | ||
| internal | ||
| libraries | ||
| metadata | ||
| paths | ||
| permissions | ||
| phases | ||
| render | ||
| resources | ||
| run | ||
| schema | ||
| scripts | ||
| tests | ||
| trampoline | ||
| bundle.go | ||
| bundle_read_only.go | ||
| bundle_test.go | ||
| context.go | ||
| context_test.go | ||
| deferred.go | ||
| deferred_test.go | ||
| if.go | ||
| if_test.go | ||
| log_string.go | ||
| mutator.go | ||
| mutator_read_only.go | ||
| mutator_test.go | ||
| parallel.go | ||
| parallel_test.go | ||
| root.go | ||
| root_test.go | ||
| seq.go | ||
| seq_test.go | ||