steve/databricks-cli
steve
/
databricks-cli
mirror of https://github.com/databricks/cli.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
databricks-cli/bundle/docsgen/output/reference.md

27 KiB
Raw Blame History

description
Configuration reference for databricks.yml

Configuration reference

This article provides reference for keys supported by Databricks Asset Bundles configuration (YAML). See _.

For complete bundle examples, see _ and the bundle-examples GitHub repository.

artifacts

Type: Map

Defines the attributes to build artifacts, where each key is the name of the artifact, and the value is a Map that defines the artifact build settings. For information about the artifacts mapping, see _.

Artifact settings defined in the top level of the bundle configuration can be overridden in the targets mapping. See _.

artifacts:
  <artifact-name>:
    <artifact-field-name>: <artifact-field-value>

:::list-table

    • Key
    • Type
    • Description
    • build
    • String
    • An optional set of non-default build commands to run locally before deployment.
    • executable
    • String
    • The executable type. Valid values are bash, sh, and cmd.
    • files
    • Sequence
    • The source files for the artifact. See _.
    • path
    • String
    • The location where the built artifact will be saved.
    • type
    • String
    • Required. The type of the artifact. Valid values are whl.

:::

Example

artifacts:
  default:
    type: whl
    build: poetry build
    path: .

artifacts.<name>.files

Type: Sequence

The source files for the artifact.

:::list-table

    • Key
    • Type
    • Description
    • source
    • String
    • Required. The path of the files used to build the artifact.

:::

bundle

Type: Map

The bundle attributes when deploying to this target,

:::list-table

    • Key
    • Type
    • Description
    • cluster_id
    • String
    • The ID of a cluster to use to run the bundle. See _.
    • compute_id
    • String
    • databricks_cli_version
    • String
    • The Databricks CLI version to use for the bundle. See _.
    • deployment
    • Map
    • The definition of the bundle deployment. For supported attributes see _. See _.
    • git
    • Map
    • The Git version control details that are associated with your bundle. For supported attributes see _. See _.
    • name
    • String
    • The name of the bundle.
    • uuid
    • String
    • Reserved. A Universally Unique Identifier (UUID) for the bundle that uniquely identifies the bundle in internal Databricks systems. This is generated when a bundle project is initialized using a Databricks template (using the databricks bundle init command).

:::

bundle.deployment

Type: Map

The definition of the bundle deployment. For supported attributes see _.

:::list-table

    • Key
    • Type
    • Description
    • fail_on_active_runs
    • Boolean
    • Whether to fail on active runs. If this is set to true a deployment that is running can be interrupted.
    • lock
    • Map
    • The deployment lock attributes. See _.

:::

bundle.deployment.lock

Type: Map

The deployment lock attributes.

:::list-table

    • Key
    • Type
    • Description
    • enabled
    • Boolean
    • Whether this lock is enabled.
    • force
    • Boolean
    • Whether to force this lock if it is enabled.

:::

bundle.git

Type: Map

The Git version control details that are associated with your bundle. For supported attributes see _.

:::list-table

    • Key
    • Type
    • Description
    • branch
    • String
    • The Git branch name. See _.
    • origin_url
    • String
    • The origin URL of the repository. See _.

:::

experimental

Type: Map

Defines attributes for experimental features.

:::list-table

    • Key
    • Type
    • Description
    • pydabs
    • Map
    • The PyDABs configuration. See _.
    • python
    • Map
    • Configures loading of Python code defined with 'databricks-bundles' package. See _.
    • python_wheel_wrapper
    • Boolean
    • Whether to use a Python wheel wrapper.
    • scripts
    • Map
    • The commands to run.
    • use_legacy_run_as
    • Boolean
    • Whether to use the legacy run_as behavior.

:::

experimental.pydabs

Type: Map

The PyDABs configuration.

:::list-table

    • Key
    • Type
    • Description
    • enabled
    • Boolean
    • Whether or not PyDABs (Private Preview) is enabled
    • import
    • Sequence
    • The PyDABs project to import to discover resources, resource generator and mutators
    • venv_path
    • String
    • The Python virtual environment path

:::

experimental.python

Type: Map

Configures loading of Python code defined with 'databricks-bundles' package.

:::list-table

    • Key
    • Type
    • Description
    • mutators
    • Sequence
    • Mutators contains a list of fully qualified function paths to mutator functions. Example: ["my_project.mutators:add_default_cluster"]
    • resources
    • Sequence
    • Resources contains a list of fully qualified function paths to load resources defined in Python code. Example: ["my_project.resources:load_resources"]
    • venv_path
    • String
    • VEnvPath is path to the virtual environment. If enabled, Python code will execute within this environment. If disabled, it defaults to using the Python interpreter available in the current shell.

:::

include

Type: Sequence

Specifies a list of path globs that contain configuration files to include within the bundle. See _

permissions

Type: Sequence

A Sequence that defines the permissions to apply to experiments, jobs, pipelines, and models defined in the bundle, where each item in the sequence is a permission for a specific entity.

See _ and _.

:::list-table

    • Key
    • Type
    • Description
    • group_name
    • String
    • The name of the group that has the permission set in level.
    • level
    • String
    • The allowed permission for user, group, service principal defined for this permission.
    • service_principal_name
    • String
    • The name of the service principal that has the permission set in level.
    • user_name
    • String
    • The name of the user that has the permission set in level.

:::

Example

permissions:
  - level: CAN_VIEW
    group_name: test-group
  - level: CAN_MANAGE
    user_name: someone@example.com
  - level: CAN_RUN
    service_principal_name: 123456-abcdef

presets

Type: Map

Defines bundle deployment presets. See _.

:::list-table

    • Key
    • Type
    • Description
    • jobs_max_concurrent_runs
    • Integer
    • The maximum concurrent runs for a job.
    • name_prefix
    • String
    • The prefix for job runs of the bundle.
    • pipelines_development
    • Boolean
    • Whether pipeline deployments should be locked in development mode.
    • source_linked_deployment
    • Boolean
    • Whether to link the deployment to the bundle source.
    • tags
    • Map
    • The tags for the bundle deployment.
    • trigger_pause_status
    • String
    • A pause status to apply to all job triggers and schedules. Valid values are PAUSED or UNPAUSED.

:::

resources

Type: Map

A Map that defines the resources for the bundle, where each key is the name of the resource, and the value is a Map that defines the resource. For more information about Databricks Asset Bundles supported resources, and resource definition reference, see _.

resources:
  <resource-type>:
    <resource-name>:
      <resource-field-name>: <resource-field-value>

:::list-table

    • Key
    • Type
    • Description
    • apps
    • Map
    • clusters
    • Map
    • The cluster definitions for the bundle, where each key is the name of a cluster. See _
    • dashboards
    • Map
    • The dashboard definitions for the bundle, where each key is the name of the dashboard. See _
    • experiments
    • Map
    • The experiment definitions for the bundle, where each key is the name of the experiment. See _
    • jobs
    • Map
    • The job definitions for the bundle, where each key is the name of the job. See _
    • model_serving_endpoints
    • Map
    • The model serving endpoint definitions for the bundle, where each key is the name of the model serving endpoint. See _
    • models
    • Map
    • The model definitions for the bundle, where each key is the name of the model. See _
    • pipelines
    • Map
    • The pipeline definitions for the bundle, where each key is the name of the pipeline. See _
    • quality_monitors
    • Map
    • The quality monitor definitions for the bundle, where each key is the name of the quality monitor. See _
    • registered_models
    • Map
    • The registered model definitions for the bundle, where each key is the name of the Unity Catalog registered model. See _
    • schemas
    • Map
    • The schema definitions for the bundle, where each key is the name of the schema. See _
    • volumes
    • Map
    • The volume definitions for the bundle, where each key is the name of the volume. See _

:::

run_as

Type: Map

The identity to use when running Databricks Asset Bundles workflows. See _.

:::list-table

    • Key
    • Type
    • Description
    • service_principal_name
    • String
    • The application ID of an active service principal. Setting this field requires the servicePrincipal/user role.
    • user_name
    • String
    • The email of an active workspace user. Non-admin users can only set this field to their own email.

:::

sync

Type: Map

The files and file paths to include or exclude in the bundle. See _.

:::list-table

    • Key
    • Type
    • Description
    • exclude
    • Sequence
    • A list of files or folders to exclude from the bundle.
    • include
    • Sequence
    • A list of files or folders to include in the bundle.
    • paths
    • Sequence
    • The local folder paths, which can be outside the bundle root, to synchronize to the workspace when the bundle is deployed.

:::

targets

Type: Map

Defines deployment targets for the bundle. See _

targets:
  <target-name>:
    <target-field-name>: <target-field-value>

:::list-table

    • Key
    • Type
    • Description
    • artifacts
    • Map
    • The artifacts to include in the target deployment. See _.
    • bundle
    • Map
    • The bundle attributes when deploying to this target. See _.
    • cluster_id
    • String
    • The ID of the cluster to use for this target.
    • compute_id
    • String
    • Deprecated. The ID of the compute to use for this target.
    • default
    • Boolean
    • Whether this target is the default target.
    • git
    • Map
    • The Git version control settings for the target. See _.
    • mode
    • String
    • The deployment mode for the target. Valid values are development or production. See _.
    • permissions
    • Sequence
    • The permissions for deploying and running the bundle in the target. See _.
    • presets
    • Map
    • The deployment presets for the target. See _.
    • resources
    • Map
    • The resource definitions for the target. See _.
    • run_as
    • Map
    • The identity to use to run the bundle, see _. See _.
    • sync
    • Map
    • The local paths to sync to the target workspace when a bundle is run or deployed. See _.
    • variables
    • Map
    • The custom variable definitions for the target. See _.
    • workspace
    • Map
    • The Databricks workspace for the target. See _.

:::

targets.<name>.artifacts

Type: Map

The artifacts to include in the target deployment.

artifacts:
  <artifact-name>:
    <artifact-field-name>: <artifact-field-value>

:::list-table

    • Key
    • Type
    • Description
    • build
    • String
    • An optional set of non-default build commands to run locally before deployment.
    • executable
    • String
    • The executable type. Valid values are bash, sh, and cmd.
    • files
    • Sequence
    • The source files for the artifact. See _.
    • path
    • String
    • The location where the built artifact will be saved.
    • type
    • String
    • Required. The type of the artifact. Valid values are whl.

:::

targets.<name>.artifacts.<name>.files

Type: Sequence

The source files for the artifact.

:::list-table

    • Key
    • Type
    • Description
    • source
    • String
    • Required. The path of the files used to build the artifact.

:::

targets.<name>.bundle

Type: Map

The bundle attributes when deploying to this target.

:::list-table

    • Key
    • Type
    • Description
    • cluster_id
    • String
    • The ID of a cluster to use to run the bundle. See _.
    • compute_id
    • String
    • databricks_cli_version
    • String
    • The Databricks CLI version to use for the bundle. See _.
    • deployment
    • Map
    • The definition of the bundle deployment. For supported attributes see _. See _.
    • git
    • Map
    • The Git version control details that are associated with your bundle. For supported attributes see _. See _.
    • name
    • String
    • The name of the bundle.
    • uuid
    • String
    • Reserved. A Universally Unique Identifier (UUID) for the bundle that uniquely identifies the bundle in internal Databricks systems. This is generated when a bundle project is initialized using a Databricks template (using the databricks bundle init command).

:::

targets.<name>.bundle.deployment

Type: Map

The definition of the bundle deployment. For supported attributes see _.

:::list-table

    • Key
    • Type
    • Description
    • fail_on_active_runs
    • Boolean
    • Whether to fail on active runs. If this is set to true a deployment that is running can be interrupted.
    • lock
    • Map
    • The deployment lock attributes. See _.

:::

targets.<name>.bundle.deployment.lock

Type: Map

The deployment lock attributes.

:::list-table

    • Key
    • Type
    • Description
    • enabled
    • Boolean
    • Whether this lock is enabled.
    • force
    • Boolean
    • Whether to force this lock if it is enabled.

:::

targets.<name>.bundle.git

Type: Map

The Git version control details that are associated with your bundle. For supported attributes see _.

:::list-table

    • Key
    • Type
    • Description
    • branch
    • String
    • The Git branch name. See _.
    • origin_url
    • String
    • The origin URL of the repository. See _.

:::

targets.<name>.git

Type: Map

The Git version control settings for the target.

:::list-table

    • Key
    • Type
    • Description
    • branch
    • String
    • The Git branch name. See _.
    • origin_url
    • String
    • The origin URL of the repository. See _.

:::

targets.<name>.permissions

Type: Sequence

The permissions for deploying and running the bundle in the target.

:::list-table

    • Key
    • Type
    • Description
    • group_name
    • String
    • The name of the group that has the permission set in level.
    • level
    • String
    • The allowed permission for user, group, service principal defined for this permission.
    • service_principal_name
    • String
    • The name of the service principal that has the permission set in level.
    • user_name
    • String
    • The name of the user that has the permission set in level.

:::

targets.<name>.presets

Type: Map

The deployment presets for the target.

:::list-table

    • Key
    • Type
    • Description
    • jobs_max_concurrent_runs
    • Integer
    • The maximum concurrent runs for a job.
    • name_prefix
    • String
    • The prefix for job runs of the bundle.
    • pipelines_development
    • Boolean
    • Whether pipeline deployments should be locked in development mode.
    • source_linked_deployment
    • Boolean
    • Whether to link the deployment to the bundle source.
    • tags
    • Map
    • The tags for the bundle deployment.
    • trigger_pause_status
    • String
    • A pause status to apply to all job triggers and schedules. Valid values are PAUSED or UNPAUSED.

:::

targets.<name>.resources

Type: Map

The resource definitions for the target.

:::list-table

    • Key
    • Type
    • Description
    • apps
    • Map
    • clusters
    • Map
    • The cluster definitions for the bundle, where each key is the name of a cluster. See _
    • dashboards
    • Map
    • The dashboard definitions for the bundle, where each key is the name of the dashboard. See _
    • experiments
    • Map
    • The experiment definitions for the bundle, where each key is the name of the experiment. See _
    • jobs
    • Map
    • The job definitions for the bundle, where each key is the name of the job. See _
    • model_serving_endpoints
    • Map
    • The model serving endpoint definitions for the bundle, where each key is the name of the model serving endpoint. See _
    • models
    • Map
    • The model definitions for the bundle, where each key is the name of the model. See _
    • pipelines
    • Map
    • The pipeline definitions for the bundle, where each key is the name of the pipeline. See _
    • quality_monitors
    • Map
    • The quality monitor definitions for the bundle, where each key is the name of the quality monitor. See _
    • registered_models
    • Map
    • The registered model definitions for the bundle, where each key is the name of the Unity Catalog registered model. See _
    • schemas
    • Map
    • The schema definitions for the bundle, where each key is the name of the schema. See _
    • volumes
    • Map
    • The volume definitions for the bundle, where each key is the name of the volume. See _

:::

targets.<name>.run_as

Type: Map

The identity to use to run the bundle, see _.

:::list-table

    • Key
    • Type
    • Description
    • service_principal_name
    • String
    • The application ID of an active service principal. Setting this field requires the servicePrincipal/user role.
    • user_name
    • String
    • The email of an active workspace user. Non-admin users can only set this field to their own email.

:::

targets.<name>.sync

Type: Map

The local paths to sync to the target workspace when a bundle is run or deployed.

:::list-table

    • Key
    • Type
    • Description
    • exclude
    • Sequence
    • A list of files or folders to exclude from the bundle.
    • include
    • Sequence
    • A list of files or folders to include in the bundle.
    • paths
    • Sequence
    • The local folder paths, which can be outside the bundle root, to synchronize to the workspace when the bundle is deployed.

:::

targets.<name>.variables

Type: Map

The custom variable definitions for the target.

variables:
  <variable-name>:
    <variable-field-name>: <variable-field-value>

:::list-table

    • Key
    • Type
    • Description
    • default
    • Any
    • description
    • String
    • The description of the variable.
    • lookup
    • Map
    • The name of the alert, cluster_policy, cluster, dashboard, instance_pool, job, metastore, pipeline, query, service_principal, or warehouse object for which to retrieve an ID. See _.
    • type
    • String
    • The type of the variable.

:::

targets.<name>.variables.<name>.lookup

Type: Map

The name of the alert, cluster_policy, cluster, dashboard, instance_pool, job, metastore, pipeline, query, service_principal, or warehouse object for which to retrieve an ID.

:::list-table

    • Key
    • Type
    • Description
    • alert
    • String
    • cluster
    • String
    • cluster_policy
    • String
    • dashboard
    • String
    • instance_pool
    • String
    • job
    • String
    • metastore
    • String
    • notification_destination
    • String
    • pipeline
    • String
    • query
    • String
    • service_principal
    • String
    • warehouse
    • String

:::

targets.<name>.workspace

Type: Map

The Databricks workspace for the target.

:::list-table

    • Key
    • Type
    • Description
    • artifact_path
    • String
    • The artifact path to use within the workspace for both deployments and workflow runs
    • auth_type
    • String
    • The authentication type.
    • azure_client_id
    • String
    • The Azure client ID
    • azure_environment
    • String
    • The Azure environment
    • azure_login_app_id
    • String
    • The Azure login app ID
    • azure_tenant_id
    • String
    • The Azure tenant ID
    • azure_use_msi
    • Boolean
    • Whether to use MSI for Azure
    • azure_workspace_resource_id
    • String
    • The Azure workspace resource ID
    • client_id
    • String
    • The client ID for the workspace
    • file_path
    • String
    • The file path to use within the workspace for both deployments and workflow runs
    • google_service_account
    • String
    • The Google service account name
    • host
    • String
    • The Databricks workspace host URL
    • profile
    • String
    • The Databricks workspace profile name
    • resource_path
    • String
    • The workspace resource path
    • root_path
    • String
    • The Databricks workspace root path
    • state_path
    • String
    • The workspace state path

:::

variables

Type: Map

Defines a custom variable for the bundle. See _.

variables:
  <variable-name>:
    <variable-field-name>: <variable-field-value>

:::list-table

    • Key
    • Type
    • Description
    • default
    • Any
    • description
    • String
    • The description of the variable
    • lookup
    • Map
    • The name of the alert, cluster_policy, cluster, dashboard, instance_pool, job, metastore, pipeline, query, service_principal, or warehouse object for which to retrieve an ID. See _.
    • type
    • String
    • The type of the variable.

:::

variables.<name>.lookup

Type: Map

The name of the alert, cluster_policy, cluster, dashboard, instance_pool, job, metastore, pipeline, query, service_principal, or warehouse object for which to retrieve an ID.

:::list-table

    • Key
    • Type
    • Description
    • alert
    • String
    • cluster
    • String
    • cluster_policy
    • String
    • dashboard
    • String
    • instance_pool
    • String
    • job
    • String
    • metastore
    • String
    • notification_destination
    • String
    • pipeline
    • String
    • query
    • String
    • service_principal
    • String
    • warehouse
    • String

:::

workspace

Type: Map

Defines the Databricks workspace for the bundle. See _.

:::list-table

    • Key
    • Type
    • Description
    • artifact_path
    • String
    • The artifact path to use within the workspace for both deployments and workflow runs
    • auth_type
    • String
    • The authentication type.
    • azure_client_id
    • String
    • The Azure client ID
    • azure_environment
    • String
    • The Azure environment
    • azure_login_app_id
    • String
    • The Azure login app ID
    • azure_tenant_id
    • String
    • The Azure tenant ID
    • azure_use_msi
    • Boolean
    • Whether to use MSI for Azure
    • azure_workspace_resource_id
    • String
    • The Azure workspace resource ID
    • client_id
    • String
    • The client ID for the workspace
    • file_path
    • String
    • The file path to use within the workspace for both deployments and workflow runs
    • google_service_account
    • String
    • The Google service account name
    • host
    • String
    • The Databricks workspace host URL
    • profile
    • String
    • The Databricks workspace profile name
    • resource_path
    • String
    • The workspace resource path
    • root_path
    • String
    • The Databricks workspace root path
    • state_path
    • String
    • The workspace state path

:::