steve/databricks-cli
steve
/
databricks-cli
mirror of https://github.com/databricks/cli.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
databricks-cli/bundle/config/resources/job.go

50 lines
1.1 KiB
Go
Raw Normal View History

Skeleton for configuration loading and mutation (#92) Load a tree of configuration files anchored at `bundle.yml` into the `config.Root` struct. All mutations (from setting defaults to merging files) are observable through the `mutator.Mutator` interface.
2022-11-18 09:57:31 +00:00
package resources
Enable environment overrides for job clusters (#658) ## Changes While they are a slice, we can identify a job cluster by its job cluster key. A job definition with multiple job clusters with the same key is always invalid. We can therefore merge definitions with the same key into one. This is compatible with how environment overrides are applied; merging a slice means appending to it. The override will end up in the job cluster slice of the original, which gives us a deterministic way to merge them. Since the alternative is an invalid configuration, this doesn't change behavior. ## Tests New test coverage.
2023-08-14 06:43:45 +00:00
import (
Added `bundle deployment bind` and `unbind` command (#1131) ## Changes Added `bundle deployment bind` and `unbind` command. This command allows to bind bundle-defined resources to existing resources in Databricks workspace so they become DABs-managed. ## Tests Manually + added E2E test
2024-02-14 18:04:45 +00:00
"context"
"strconv"
Make resource and artifact paths in bundle config relative to config folder (#708) # Warning: breaking change ## Changes Instead of having paths in bundle config files be relative to bundle root even if the config file is nested, this PR makes such paths relative to the folder where the config is located. When bundle is initialised, these paths will be transformed to relative paths based on bundle root. For example, we have file structure like this ``` - mybundle | - bundle.yml | - subfolder | -- resource.yml | -- my.whl ``` Previously, we had to reference `my.whl` in resource.yml like this, which was confusing because resource.yml is in the same subfolder ``` sync: include: - ./subfolder/*.whl ... tasks: - task_key: name libraries: - whl: ./subfolder/my.whl ... ``` After the change we can reference it like this (which is in line with the current behaviour for notebooks) ``` sync: include: - ./*.whl ... tasks: - task_key: name libraries: - whl: ./my.whl ... ``` ## Tests Existing `translate_path_tests` successfully passed after refactoring. Added a couple of uses cases for `Libraries` paths. Added a bundle config tests with include config and sync section --------- Co-authored-by: Pieter Noordhuis <pieter.noordhuis@databricks.com>
2023-09-04 09:55:01 +00:00
"github.com/databricks/cli/bundle/config/paths"
Added `bundle deployment bind` and `unbind` command (#1131) ## Changes Added `bundle deployment bind` and `unbind` command. This command allows to bind bundle-defined resources to existing resources in Databricks workspace so they become DABs-managed. ## Tests Manually + added E2E test
2024-02-14 18:04:45 +00:00
"github.com/databricks/cli/libs/log"
"github.com/databricks/databricks-sdk-go"
Update Go SDK to 0.23.0 and use custom marshaller (#772) ## Changes Update Go SDK to 0.23.0 and use custom marshaller. ## Tests * Run unit tests * Run nightly * Manual test: ``` ./cli jobs create --json @myjob.json ``` with ``` { "name": "my-job-marshal-test-go", "tasks": [{ "task_key": "testgomarshaltask", "new_cluster": { "num_workers": 0, "spark_version": "10.4.x-scala2.12", "node_type_id": "Standard_DS3_v2" }, "libraries": [ { "jar": "dbfs:/max/jars/exampleJarTask.jar" } ], "spark_jar_task": { "main_class_name": "com.databricks.quickstart.exampleTask" } }] } ``` Main branch: ``` Error: Cluster validation error: Missing required field: settings.cluster_spec.new_cluster.size ``` This branch: ``` { "job_id":<jobid> } ``` --------- Co-authored-by: Miles Yucht <miles@databricks.com>
2023-10-16 06:56:06 +00:00
"github.com/databricks/databricks-sdk-go/marshal"
Enable environment overrides for job clusters (#658) ## Changes While they are a slice, we can identify a job cluster by its job cluster key. A job definition with multiple job clusters with the same key is always invalid. We can therefore merge definitions with the same key into one. This is compatible with how environment overrides are applied; merging a slice means appending to it. The override will end up in the job cluster slice of the original, which gives us a deterministic way to merge them. Since the alternative is an invalid configuration, this doesn't change behavior. ## Tests New test coverage.
2023-08-14 06:43:45 +00:00
"github.com/databricks/databricks-sdk-go/service/jobs"
)
Add job and pipeline structs (#94)
2022-11-18 10:12:24 +00:00
Revert "Rename jobs -> workflows" (#118) This reverts PR #111. This reverts commit 230811031f54f4c9e90524a0a1863ce628f6d954.
2022-12-01 21:39:15 +00:00
type Job struct {
Add "bundle summary" command (#1123) The plan is to use the new command in the Databricks VSCode extension to render "modified" UI state in the bundle resource tree elements, plus use resource IDs to generate links for the resources ### New revision - Renamed `remote-state` to `summary` - Added "modified statuses" to all resources. Currently we don't set "updated" status - it's either nothing, or created/deleted - Added tests for the `TerraformToBundle` command
2024-01-25 11:32:47 +00:00
ID string `json:"id,omitempty" bundle:"readonly"`
Permissions []Permission `json:"permissions,omitempty"`
ModifiedStatus ModifiedStatus `json:"modified_status,omitempty" bundle:"internal"`
Add job and pipeline structs (#94)
2022-11-18 10:12:24 +00:00
Make resource and artifact paths in bundle config relative to config folder (#708) # Warning: breaking change ## Changes Instead of having paths in bundle config files be relative to bundle root even if the config file is nested, this PR makes such paths relative to the folder where the config is located. When bundle is initialised, these paths will be transformed to relative paths based on bundle root. For example, we have file structure like this ``` - mybundle | - bundle.yml | - subfolder | -- resource.yml | -- my.whl ``` Previously, we had to reference `my.whl` in resource.yml like this, which was confusing because resource.yml is in the same subfolder ``` sync: include: - ./subfolder/*.whl ... tasks: - task_key: name libraries: - whl: ./subfolder/my.whl ... ``` After the change we can reference it like this (which is in line with the current behaviour for notebooks) ``` sync: include: - ./*.whl ... tasks: - task_key: name libraries: - whl: ./my.whl ... ``` ## Tests Existing `translate_path_tests` successfully passed after refactoring. Added a couple of uses cases for `Libraries` paths. Added a bundle config tests with include config and sync section --------- Co-authored-by: Pieter Noordhuis <pieter.noordhuis@databricks.com>
2023-09-04 09:55:01 +00:00
paths.Paths
Store relative path to configuration file for every resource (#322) ## Changes If a configuration file is located in a subdirectory of the bundle root, files referenced from that configuration file should be relative to its configuration file's directory instead of the bundle root. ## Tests * New tests in `bundle/config/mutator/translate_paths_test.go`. * Existing tests under `bundle/tests` pass and are augmented to assert on paths. --------- Co-authored-by: shreyas-goenka <88374338+shreyas-goenka@users.noreply.github.com>
2023-04-12 14:17:13 +00:00
Add job and pipeline structs (#94)
2022-11-18 10:12:24 +00:00
*jobs.JobSettings
Skeleton for configuration loading and mutation (#92) Load a tree of configuration files anchored at `bundle.yml` into the `config.Root` struct. All mutations (from setting defaults to merging files) are observable through the `mutator.Mutator` interface.
2022-11-18 09:57:31 +00:00
}
Enable environment overrides for job clusters (#658) ## Changes While they are a slice, we can identify a job cluster by its job cluster key. A job definition with multiple job clusters with the same key is always invalid. We can therefore merge definitions with the same key into one. This is compatible with how environment overrides are applied; merging a slice means appending to it. The override will end up in the job cluster slice of the original, which gives us a deterministic way to merge them. Since the alternative is an invalid configuration, this doesn't change behavior. ## Tests New test coverage.
2023-08-14 06:43:45 +00:00
Update Go SDK to 0.23.0 and use custom marshaller (#772) ## Changes Update Go SDK to 0.23.0 and use custom marshaller. ## Tests * Run unit tests * Run nightly * Manual test: ``` ./cli jobs create --json @myjob.json ``` with ``` { "name": "my-job-marshal-test-go", "tasks": [{ "task_key": "testgomarshaltask", "new_cluster": { "num_workers": 0, "spark_version": "10.4.x-scala2.12", "node_type_id": "Standard_DS3_v2" }, "libraries": [ { "jar": "dbfs:/max/jars/exampleJarTask.jar" } ], "spark_jar_task": { "main_class_name": "com.databricks.quickstart.exampleTask" } }] } ``` Main branch: ``` Error: Cluster validation error: Missing required field: settings.cluster_spec.new_cluster.size ``` This branch: ``` { "job_id":<jobid> } ``` --------- Co-authored-by: Miles Yucht <miles@databricks.com>
2023-10-16 06:56:06 +00:00
func (s *Job) UnmarshalJSON(b []byte) error {
return marshal.Unmarshal(b, s)
}
func (s Job) MarshalJSON() ([]byte, error) {
return marshal.Marshal(s)
}
Added `bundle deployment bind` and `unbind` command (#1131) ## Changes Added `bundle deployment bind` and `unbind` command. This command allows to bind bundle-defined resources to existing resources in Databricks workspace so they become DABs-managed. ## Tests Manually + added E2E test
2024-02-14 18:04:45 +00:00
func (j *Job) Exists(ctx context.Context, w *databricks.WorkspaceClient, id string) (bool, error) {
jobId, err := strconv.Atoi(id)
if err != nil {
return false, err
}
_, err = w.Jobs.Get(ctx, jobs.GetJobRequest{
JobId: int64(jobId),
})
if err != nil {
log.Debugf(ctx, "job %s does not exist", id)
return false, err
}
return true, nil
}
func (j *Job) TerraformResourceName() string {
return "databricks_job"
}