databricks-cli/bundle/config/lock.go

31 lines
938 B
Go
Raw Permalink Normal View History

package config
type Lock struct {
Disable locking for development mode (#1302) ## Changes This changes `databricks bundle deploy` so that it skips the lock acquisition/release step for a `mode: development` target: * This saves about 2 seconds (measured over 100 runs on a quiet/busy workspace). * This helps avoid the `deploy lock acquired by lennart@company.com at 2024-02-28 15:48:38.40603 +0100 CET. Use --force-lock to override` error * Risk: this may cause deployment conflicts, but since dev mode deployments are always scoped to a user, that risk should be minimal Update after discussion: * This behavior can now be disabled via a setting. * Docs PR: https://github.com/databricks/docs/pull/15873 ## Measurements ### 100 deployments of the "python_default" project to an empty workspace _Before this branch:_ p50 time: 11.479 seconds p90 time: 11.757 seconds _After this branch:_ p50 time: 9.386 seconds p90 time: 9.599 seconds ### 100 deployments of the "python_default" project to a busy (staging) workspace _Before this branch:_ * p50 time: 13.335 seconds * p90 time: 15.295 seconds _After this branch:_ * p50 time: 11.397 seconds * p90 time: 11.743 seconds ### Typical duration of deployment steps * Acquiring Deployment Lock: 1.096 seconds * Deployment Preparations and Operations: 1.477 seconds * Uploading Artifacts: 1.26 seconds * Finalizing Deployment: 9.699 seconds * Releasing Deployment Lock: 1.198 seconds --------- Co-authored-by: Pieter Noordhuis <pcnoordhuis@gmail.com> Co-authored-by: Andrew Nester <andrew.nester.dev@gmail.com>
2024-04-18 01:59:39 +00:00
// Enabled toggles deployment lock. True by default except in development mode.
// Use a pointer value so that only explicitly configured values are set
// and we don't merge configuration with zero-initialized values.
Persist deployment metadata in WSFS (#845) ## Changes This PR introduces a metadata struct that stores a subset of bundle configuration that we wish to expose to other Databricks services that wish to integrate with bundles. This metadata file is uploaded to a file `${bundle.workspace.state_path}/metadata.json` in the WSFS destination of the bundle deployment. Documentation for emitted metadata fields: * `version`: Version for the metadata file schema * `config.bundle.git.branch`: Name of the git branch the bundle was deployed from. * `config.bundle.git.origin_url`: URL for git remote "origin" * `config.bundle.git.bundle_root_path`: Relative path of the bundle root from the root of the git repository. Is set to "." if they are the same. * `config.bundle.git.commit`: SHA-1 commit hash of the exact commit this bundle was deployed from. Note, the deployment might not exactly match this commit version if there are changes that have not been committed to git at deploy time, * `file_path`: Path in workspace where we sync bundle files to. * `resources.jobs.[job-ref].id`: Id of the job * `resources.jobs.[job-ref].relative_path`: Relative path of the yaml config file from the bundle root where this job was defined. Example metadata object when bundle root and git root are the same: ```json { "version": 1, "config": { "bundle": { "lock": {}, "git": { "branch": "master", "origin_url": "www.host.com", "commit": "7af8e5d3f5dceffff9295d42d21606ccf056dce0", "bundle_root_path": "." } }, "workspace": { "file_path": "/Users/shreyas.goenka@databricks.com/.bundle/pipeline-progress/default/files" }, "resources": { "jobs": { "bar": { "id": "245921165354846", "relative_path": "databricks.yml" } } }, "sync": {} } } ``` Example metadata when the git root is one level above the bundle repo: ```json { "version": 1, "config": { "bundle": { "lock": {}, "git": { "branch": "dev-branch", "origin_url": "www.my-repo.com", "commit": "3db46ef750998952b00a2b3e7991e31787e4b98b", "bundle_root_path": "pipeline-progress" } }, "workspace": { "file_path": "/Users/shreyas.goenka@databricks.com/.bundle/pipeline-progress/default/files" }, "resources": { "jobs": { "bar": { "id": "245921165354846", "relative_path": "databricks.yml" } } }, "sync": {} } } ``` This unblocks integration to the jobs break glass UI for bundles. ## Tests Unit tests and integration tests.
2023-10-27 12:55:43 +00:00
Enabled *bool `json:"enabled,omitempty"`
// Force acquisition of deployment lock even if it is currently held.
// This may be necessary if a prior deployment failed to release the lock.
Persist deployment metadata in WSFS (#845) ## Changes This PR introduces a metadata struct that stores a subset of bundle configuration that we wish to expose to other Databricks services that wish to integrate with bundles. This metadata file is uploaded to a file `${bundle.workspace.state_path}/metadata.json` in the WSFS destination of the bundle deployment. Documentation for emitted metadata fields: * `version`: Version for the metadata file schema * `config.bundle.git.branch`: Name of the git branch the bundle was deployed from. * `config.bundle.git.origin_url`: URL for git remote "origin" * `config.bundle.git.bundle_root_path`: Relative path of the bundle root from the root of the git repository. Is set to "." if they are the same. * `config.bundle.git.commit`: SHA-1 commit hash of the exact commit this bundle was deployed from. Note, the deployment might not exactly match this commit version if there are changes that have not been committed to git at deploy time, * `file_path`: Path in workspace where we sync bundle files to. * `resources.jobs.[job-ref].id`: Id of the job * `resources.jobs.[job-ref].relative_path`: Relative path of the yaml config file from the bundle root where this job was defined. Example metadata object when bundle root and git root are the same: ```json { "version": 1, "config": { "bundle": { "lock": {}, "git": { "branch": "master", "origin_url": "www.host.com", "commit": "7af8e5d3f5dceffff9295d42d21606ccf056dce0", "bundle_root_path": "." } }, "workspace": { "file_path": "/Users/shreyas.goenka@databricks.com/.bundle/pipeline-progress/default/files" }, "resources": { "jobs": { "bar": { "id": "245921165354846", "relative_path": "databricks.yml" } } }, "sync": {} } } ``` Example metadata when the git root is one level above the bundle repo: ```json { "version": 1, "config": { "bundle": { "lock": {}, "git": { "branch": "dev-branch", "origin_url": "www.my-repo.com", "commit": "3db46ef750998952b00a2b3e7991e31787e4b98b", "bundle_root_path": "pipeline-progress" } }, "workspace": { "file_path": "/Users/shreyas.goenka@databricks.com/.bundle/pipeline-progress/default/files" }, "resources": { "jobs": { "bar": { "id": "245921165354846", "relative_path": "databricks.yml" } } }, "sync": {} } } ``` This unblocks integration to the jobs break glass UI for bundles. ## Tests Unit tests and integration tests.
2023-10-27 12:55:43 +00:00
Force bool `json:"force,omitempty"`
}
Disable locking for development mode (#1302) ## Changes This changes `databricks bundle deploy` so that it skips the lock acquisition/release step for a `mode: development` target: * This saves about 2 seconds (measured over 100 runs on a quiet/busy workspace). * This helps avoid the `deploy lock acquired by lennart@company.com at 2024-02-28 15:48:38.40603 +0100 CET. Use --force-lock to override` error * Risk: this may cause deployment conflicts, but since dev mode deployments are always scoped to a user, that risk should be minimal Update after discussion: * This behavior can now be disabled via a setting. * Docs PR: https://github.com/databricks/docs/pull/15873 ## Measurements ### 100 deployments of the "python_default" project to an empty workspace _Before this branch:_ p50 time: 11.479 seconds p90 time: 11.757 seconds _After this branch:_ p50 time: 9.386 seconds p90 time: 9.599 seconds ### 100 deployments of the "python_default" project to a busy (staging) workspace _Before this branch:_ * p50 time: 13.335 seconds * p90 time: 15.295 seconds _After this branch:_ * p50 time: 11.397 seconds * p90 time: 11.743 seconds ### Typical duration of deployment steps * Acquiring Deployment Lock: 1.096 seconds * Deployment Preparations and Operations: 1.477 seconds * Uploading Artifacts: 1.26 seconds * Finalizing Deployment: 9.699 seconds * Releasing Deployment Lock: 1.198 seconds --------- Co-authored-by: Pieter Noordhuis <pcnoordhuis@gmail.com> Co-authored-by: Andrew Nester <andrew.nester.dev@gmail.com>
2024-04-18 01:59:39 +00:00
// IsEnabled checks if the deployment lock is enabled.
func (lock Lock) IsEnabled() bool {
if lock.Enabled != nil {
return *lock.Enabled
}
return true
}
Disable locking for development mode (#1302) ## Changes This changes `databricks bundle deploy` so that it skips the lock acquisition/release step for a `mode: development` target: * This saves about 2 seconds (measured over 100 runs on a quiet/busy workspace). * This helps avoid the `deploy lock acquired by lennart@company.com at 2024-02-28 15:48:38.40603 +0100 CET. Use --force-lock to override` error * Risk: this may cause deployment conflicts, but since dev mode deployments are always scoped to a user, that risk should be minimal Update after discussion: * This behavior can now be disabled via a setting. * Docs PR: https://github.com/databricks/docs/pull/15873 ## Measurements ### 100 deployments of the "python_default" project to an empty workspace _Before this branch:_ p50 time: 11.479 seconds p90 time: 11.757 seconds _After this branch:_ p50 time: 9.386 seconds p90 time: 9.599 seconds ### 100 deployments of the "python_default" project to a busy (staging) workspace _Before this branch:_ * p50 time: 13.335 seconds * p90 time: 15.295 seconds _After this branch:_ * p50 time: 11.397 seconds * p90 time: 11.743 seconds ### Typical duration of deployment steps * Acquiring Deployment Lock: 1.096 seconds * Deployment Preparations and Operations: 1.477 seconds * Uploading Artifacts: 1.26 seconds * Finalizing Deployment: 9.699 seconds * Releasing Deployment Lock: 1.198 seconds --------- Co-authored-by: Pieter Noordhuis <pcnoordhuis@gmail.com> Co-authored-by: Andrew Nester <andrew.nester.dev@gmail.com>
2024-04-18 01:59:39 +00:00
// IsExplicitlyEnabled checks if the deployment lock is explicitly enabled.
// Only returns true if locking is explicitly set using a command-line
// flag or configuration file.
func (lock Lock) IsExplicitlyEnabled() bool {
if lock.Enabled != nil {
return *lock.Enabled
}
return false
}