steve/databricks-cli
steve
/
databricks-cli
mirror of https://github.com/databricks/cli.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
databricks-cli/bundle/config/mutator/mutator.go

36 lines
1.1 KiB
Go
Raw Normal View History

Skeleton for configuration loading and mutation (#92) Load a tree of configuration files anchored at `bundle.yml` into the `config.Root` struct. All mutations (from setting defaults to merging files) are observable through the `mutator.Mutator` interface.
2022-11-18 09:57:31 +00:00
package mutator
Move mutator interface to top level bundle package (#105) While working on artifact upload and workspace interrogation I realized this mutator interface needs to: 1. Operate at the whole bundle level so it can apply to both configuration and internal state 2. Include a `context.Context` parameter for a) long running operations and b) progress reporting Previous interface: ``` Apply(*config.Root) ([]Mutator, error) ``` New interface: ``` Apply(context.Context, *Bundle) ([]Mutator, error) ```
2022-11-28 09:59:43 +00:00
import (
Rename bricks -> databricks (#389) ## Changes Rename all instances of "bricks" to "databricks". ## Tests * Confirmed the goreleaser build works, uses the correct new binary name, and produces the right archives. * Help output is confirmed to be correct. * Output of `git grep -w bricks` is minimal with a couple changes remaining for after the repository rename.
2023-05-16 16:35:39 +00:00
"github.com/databricks/cli/bundle"
Added support for experimental scripts section (#632) ## Changes Added support for experimental scripts section It allows execution of arbitrary bash commands during certain bundle lifecycle steps. ## Tests Example of configuration ```yaml bundle: name: wheel-task workspace: host: *** experimental: scripts: prebuild: | echo 'Prebuild 1' echo 'Prebuild 2' postbuild: "echo 'Postbuild 1' && echo 'Postbuild 2'" predeploy: | echo 'Checking go version...' go version postdeploy: | echo 'Checking python version...' python --version resources: jobs: test_job: name: "[${bundle.environment}] My Wheel Job" tasks: - task_key: TestTask existing_cluster_id: "***" python_wheel_task: package_name: "my_test_code" entry_point: "run" libraries: - whl: ./dist/*.whl ``` Output ```bash andrew.nester@HFW9Y94129 wheel % databricks bundle deploy artifacts.whl.AutoDetect: Detecting Python wheel project... artifacts.whl.AutoDetect: Found Python wheel project at /Users/andrew.nester/dabs/wheel 'Prebuild 1' 'Prebuild 2' artifacts.whl.Build(my_test_code): Building... artifacts.whl.Build(my_test_code): Build succeeded 'Postbuild 1' 'Postbuild 2' 'Checking go version...' go version go1.19.9 darwin/arm64 Starting upload of bundle files Uploaded bundle files at /Users/andrew.nester@databricks.com/.bundle/wheel-task/default/files! artifacts.Upload(my_test_code-0.0.0a0-py3-none-any.whl): Uploading... artifacts.Upload(my_test_code-0.0.0a0-py3-none-any.whl): Upload succeeded Starting resource deployment Resource deployment completed! 'Checking python version...' Python 2.7.18 ```
2023-09-14 10:14:13 +00:00
"github.com/databricks/cli/bundle/config"
Load bundle configuration from mutator (#1318) ## Changes Prior to this change, the bundle configuration entry point was loaded from the function `bundle.Load`. Other configuration files were only loaded once the caller applied the first set of mutators. This separation was unnecessary and not ideal in light of gathering diagnostics while loading _any_ configuration file, not just the ones from the includes. This change: * Updates `bundle.Load` to only verify that the specified path is a valid bundle root. * Moves mutators that perform loading to `bundle/config/loader`. * Adds a "load" phase that takes the place of applying `DefaultMutators`. Follow ups: * Rename `bundle.Load` -> `bundle.Find` (because it no longer performs loading) This change depends on #1316 and #1317. ## Tests Tests pass.
2024-03-27 10:49:05 +00:00
"github.com/databricks/cli/bundle/config/loader"
Add ApplyPythonMutator (#1430) ## Changes Add ApplyPythonMutator, which will fork the Python subprocess and process pipe bundle configuration through it. It's enabled through `experimental` section, for example: ```yaml experimental: pydabs: enable: true venv_path: .venv ``` For now, it's limited to two phases in the mutator pipeline: - `load`: adds new jobs - `init`: adds new jobs, or modifies existing ones It's enforced that no jobs are modified in `load` and not jobs are deleted in `load/init`, because, otherwise, it will break existing assumptions. ## Tests Unit tests
2024-06-20 08:43:08 +00:00
pythonmutator "github.com/databricks/cli/bundle/config/mutator/python"
Use dynamic walking to validate unique resource keys (#1614) ## Changes This PR: 1. Uses dynamic walking (via the `dyn.MapByPattern` func) to validate no two resources have the same resource key. The allows us to remove this validation at merge time. 2. Modifies `dyn.Mapping` to always return a sorted slice of pairs. This makes traversal functions like `dyn.Walk` or `dyn.MapByPattern` deterministic. ## Tests Unit tests. Also manually.
2024-07-29 13:04:02 +00:00
"github.com/databricks/cli/bundle/config/validate"
Added support for experimental scripts section (#632) ## Changes Added support for experimental scripts section It allows execution of arbitrary bash commands during certain bundle lifecycle steps. ## Tests Example of configuration ```yaml bundle: name: wheel-task workspace: host: *** experimental: scripts: prebuild: | echo 'Prebuild 1' echo 'Prebuild 2' postbuild: "echo 'Postbuild 1' && echo 'Postbuild 2'" predeploy: | echo 'Checking go version...' go version postdeploy: | echo 'Checking python version...' python --version resources: jobs: test_job: name: "[${bundle.environment}] My Wheel Job" tasks: - task_key: TestTask existing_cluster_id: "***" python_wheel_task: package_name: "my_test_code" entry_point: "run" libraries: - whl: ./dist/*.whl ``` Output ```bash andrew.nester@HFW9Y94129 wheel % databricks bundle deploy artifacts.whl.AutoDetect: Detecting Python wheel project... artifacts.whl.AutoDetect: Found Python wheel project at /Users/andrew.nester/dabs/wheel 'Prebuild 1' 'Prebuild 2' artifacts.whl.Build(my_test_code): Building... artifacts.whl.Build(my_test_code): Build succeeded 'Postbuild 1' 'Postbuild 2' 'Checking go version...' go version go1.19.9 darwin/arm64 Starting upload of bundle files Uploaded bundle files at /Users/andrew.nester@databricks.com/.bundle/wheel-task/default/files! artifacts.Upload(my_test_code-0.0.0a0-py3-none-any.whl): Uploading... artifacts.Upload(my_test_code-0.0.0a0-py3-none-any.whl): Upload succeeded Starting resource deployment Resource deployment completed! 'Checking python version...' Python 2.7.18 ```
2023-09-14 10:14:13 +00:00
"github.com/databricks/cli/bundle/scripts"
Move mutator interface to top level bundle package (#105) While working on artifact upload and workspace interrogation I realized this mutator interface needs to: 1. Operate at the whole bundle level so it can apply to both configuration and internal state 2. Include a `context.Context` parameter for a) long running operations and b) progress reporting Previous interface: ``` Apply(*config.Root) ([]Mutator, error) ``` New interface: ``` Apply(context.Context, *Bundle) ([]Mutator, error) ```
2022-11-28 09:59:43 +00:00
)
Skeleton for configuration loading and mutation (#92) Load a tree of configuration files anchored at `bundle.yml` into the `config.Root` struct. All mutations (from setting defaults to merging files) are observable through the `mutator.Mutator` interface.
2022-11-18 09:57:31 +00:00
Move mutator interface to top level bundle package (#105) While working on artifact upload and workspace interrogation I realized this mutator interface needs to: 1. Operate at the whole bundle level so it can apply to both configuration and internal state 2. Include a `context.Context` parameter for a) long running operations and b) progress reporting Previous interface: ``` Apply(*config.Root) ([]Mutator, error) ``` New interface: ``` Apply(context.Context, *Bundle) ([]Mutator, error) ```
2022-11-28 09:59:43 +00:00
func DefaultMutators() []bundle.Mutator {
return []bundle.Mutator{
Execute preinit after entry point to make sure scripts are loaded (#1351) ## Changes Execute preinit after entry point to make sure scripts are loaded
2024-04-08 14:32:21 +00:00
loader.EntryPoint(),
// Execute preinit script before processing includes.
Fixed pre-init script order (#1348) ## Changes `preinit` script needs to be executed before processing configuration files to allow the script to modify the configuration or add own configuration files.
2024-04-08 13:28:38 +00:00
// It needs to be done before processing configuration files to allow
// the script to modify the configuration or add own configuration files.
scripts.Execute(config.ScriptPreInit),
Load bundle configuration from mutator (#1318) ## Changes Prior to this change, the bundle configuration entry point was loaded from the function `bundle.Load`. Other configuration files were only loaded once the caller applied the first set of mutators. This separation was unnecessary and not ideal in light of gathering diagnostics while loading _any_ configuration file, not just the ones from the includes. This change: * Updates `bundle.Load` to only verify that the specified path is a valid bundle root. * Moves mutators that perform loading to `bundle/config/loader`. * Adds a "load" phase that takes the place of applying `DefaultMutators`. Follow ups: * Rename `bundle.Load` -> `bundle.Find` (because it no longer performs loading) This change depends on #1316 and #1317. ## Tests Tests pass.
2024-03-27 10:49:05 +00:00
loader.ProcessRootIncludes(),
Allow specifying CLI version constraints required to run the bundle (#1320) ## Changes Allow specifying CLI version constraints required to run the bundle Example of configuration: #### only allow specific version ``` bundle: name: my-bundle databricks_cli_version: "0.210.0" ``` #### allow all patch releases ``` bundle: name: my-bundle databricks_cli_version: "0.210.*" ``` #### constrain minimum version ``` bundle: name: my-bundle databricks_cli_version: ">= 0.210.0" ``` #### constrain range ``` bundle: name: my-bundle databricks_cli_version: ">= 0.210.0, <= 1.0.0" ``` For other examples see: https://github.com/Masterminds/semver?tab=readme-ov-file#checking-version-constraints Example error ``` sh-3.2$ databricks bundle validate Error: Databricks CLI version constraint not satisfied. Required: >= 1.0.0, current: 0.216.0 ``` ## Tests Added unit test cover all possible configuration permutations --------- Co-authored-by: Lennart Kats (databricks) <lennart.kats@databricks.com>
2024-04-02 12:55:21 +00:00
// Verify that the CLI version is within the specified range.
VerifyCliVersion(),
Use dynamic configuration model in bundles (#1098) ## Changes This is a fundamental change to how we load and process bundle configuration. We now depend on the configuration being represented as a `dyn.Value`. This representation is functionally equivalent to Go's `any` (it is variadic) and allows us to capture metadata associated with a value, such as where it was defined (e.g. file, line, and column). It also allows us to represent Go's zero values properly (e.g. empty string, integer equal to 0, or boolean false). Using this representation allows us to let the configuration model deviate from the typed structure we have been relying on so far (`config.Root`). We need to deviate from these types when using variables for fields that are not a string themselves. For example, using `${var.num_workers}` for an integer `workers` field was impossible until now (though not implemented in this change). The loader for a `dyn.Value` includes functionality to capture any and all type mismatches between the user-defined configuration and the expected types. These mismatches can be surfaced as validation errors in future PRs. Given that many mutators expect the typed struct to be the source of truth, this change converts between the dynamic representation and the typed representation on mutator entry and exit. Existing mutators can continue to modify the typed representation and these modifications are reflected in the dynamic representation (see `MarkMutatorEntry` and `MarkMutatorExit` in `bundle/config/root.go`). Required changes included in this change: * The existing interpolation package is removed in favor of `libs/dyn/dynvar`. * Functionality to merge job clusters, job tasks, and pipeline clusters are now all broken out into their own mutators. To be implemented later: * Allow variable references for non-string types. * Surface diagnostics about the configuration provided by the user in the validation output. * Some mutators use a resource's configuration file path to resolve related relative paths. These depend on `bundle/config/paths.Path` being set and populated through `ConfigureConfigFilePath`. Instead, they should interact with the dynamically typed configuration directly. Doing this also unlocks being able to differentiate different base paths used within a job (e.g. a task override with a relative path defined in a directory other than the base job). ## Tests * Existing unit tests pass (some have been modified to accommodate) * Integration tests pass
2024-02-16 19:41:58 +00:00
EnvironmentsToTargets(),
Added support for creating all-purpose clusters (#1698) ## Changes Added support for creating all-purpose clusters Example of configuration ``` bundle: name: clusters resources: clusters: test_cluster: cluster_name: "Test Cluster" num_workers: 2 node_type_id: "i3.xlarge" autoscale: min_workers: 2 max_workers: 7 spark_version: "13.3.x-scala2.12" spark_conf: "spark.executor.memory": "2g" jobs: test_job: name: "Test Job" tasks: - task_key: test_task existing_cluster_id: ${resources.clusters.test_cluster.id} notebook_task: notebook_path: "./src/test.py" targets: development: mode: development compute_id: ${resources.clusters.test_cluster.id} ``` ## Tests Added unit, config and E2E tests
2024-09-23 10:42:34 +00:00
ComputeIdToClusterId(),
Initialize variable definitions that are defined without properties (#966) ## Changes We can debate whether or not variable definitions without properties are valid, but in no case should this panic the CLI. Fixes #934. ## Tests Unit.
2023-11-08 11:01:14 +00:00
InitializeVariables(),
Renamed `environments` to `targets` in bundle configuration (#670) ## Changes Renamed Environments to Targets in bundle.yml. The change is backward-compatible and customers can continue to use `environments` in the time being. ## Tests Added tests which checks that both `environments` and `targets` sections in bundle.yml works correctly
2023-08-17 15:22:32 +00:00
DefineDefaultTarget(),
PythonMutator: replace stdin/stdout with files (#1512) ## Changes Replace stdin/stdout with files in `PythonMutator`. Files are created in a temporary directory. Rename `ApplyPythonMutator` to `PythonMutator`. Add test for `dyn.Location` behavior during the "load" stage. ## Tests Unit tests
2024-06-24 07:47:41 +00:00
pythonmutator.PythonMutator(pythonmutator.PythonMutatorPhaseLoad),
Use dynamic walking to validate unique resource keys (#1614) ## Changes This PR: 1. Uses dynamic walking (via the `dyn.MapByPattern` func) to validate no two resources have the same resource key. The allows us to remove this validation at merge time. 2. Modifies `dyn.Mapping` to always return a sorted slice of pairs. This makes traversal functions like `dyn.Walk` or `dyn.MapByPattern` deterministic. ## Tests Unit tests. Also manually.
2024-07-29 13:04:02 +00:00
// Note: This mutator must run before the target overrides are merged.
// See the mutator for more details.
validate.UniqueResourceKeys(),
Skeleton for configuration loading and mutation (#92) Load a tree of configuration files anchored at `bundle.yml` into the `config.Root` struct. All mutations (from setting defaults to merging files) are observable through the `mutator.Mutator` interface.
2022-11-18 09:57:31 +00:00
}
}