steve/databricks-cli
steve
/
databricks-cli
mirror of https://github.com/databricks/cli.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
databricks-cli/bundle/tests/job_and_pipeline/databricks.yml

Ignoring revisions in .git-blame-ignore-revs. Click here to bypass and see the normal blame view.

45 lines
936 B
YAML
Raw Normal View History

Add job and pipeline structs (#94)
2022-11-18 10:12:24 +00:00
resources:
pipelines:
nyc_taxi_pipeline:
name: "nyc taxi loader"
libraries:
- notebook:
path: ./dlt/nyc_taxi_loader
Renamed `environments` to `targets` in bundle configuration (#670) ## Changes Renamed Environments to Targets in bundle.yml. The change is backward-compatible and customers can continue to use `environments` in the time being. ## Tests Added tests which checks that both `environments` and `targets` sections in bundle.yml works correctly
2023-08-17 15:22:32 +00:00
targets:
Add job and pipeline structs (#94)
2022-11-18 10:12:24 +00:00
development:
Add development runs (#522) This implements the "development run" functionality that we desire for DABs in the workspace / IDE. ## bundle.yml changes In bundle.yml, there should be a "dev" environment that is marked as `mode: debug`: ``` environments: dev: default: true mode: development # future accepted values might include pull_request, production ``` Setting `mode` to `development` indicates that this environment is used just for running things for development. This results in several changes to deployed assets: * All assets will get '[dev]' in their name and will get a 'dev' tag * All assets will be hidden from the list of assets (future work; e.g. for jobs we would have a special job_type that hides it from the list) * All deployed assets will be ephemeral (future work, we need some form of garbage collection) * Pipelines will be marked as 'development: true' * Jobs can run on development compute through the `--compute` parameter in the CLI * Jobs get their schedule / triggers paused * Jobs get concurrent runs (it's really annoying if your runs get skipped because the last run was still in progress) Other accepted values for `mode` are `default` (which does nothing) and `pull-request` (which is reserved for future use). ## CLI changes To run a single job called "shark_sighting" on existing compute, use the following commands: ``` $ databricks bundle deploy --compute 0617-201942-9yd9g8ix $ databricks bundle run shark_sighting ``` which would deploy and run a job called "[dev] shark_sightings" on the compute provided. Note that `--compute` is not accepted in production environments, so we show an error if `mode: development` is not used. The `run --deploy` command offers a convenient shorthand for the common combination of deploying & running: ``` $ export DATABRICKS_COMPUTE=0617-201942-9yd9g8ix $ bundle run --deploy shark_sightings ``` The `--deploy` addition isn't really essential and I welcome feedback 🤔 I played with the idea of a "debug" or "dev" command but that seemed to only make the option space even broader for users. The above could work well with an IDE or workspace that automatically sets the target compute. One more thing I added is`run --no-wait` can now be used to run something without waiting for it to be completed (useful for IDE-like environments that can display progress themselves). ``` $ bundle run --deploy shark_sightings --no-wait ```
2023-07-12 06:51:54 +00:00
mode: development
Add job and pipeline structs (#94)
2022-11-18 10:12:24 +00:00
resources:
pipelines:
nyc_taxi_pipeline:
target: nyc_taxi_development
development: true
staging:
resources:
pipelines:
nyc_taxi_pipeline:
target: nyc_taxi_staging
development: false
production:
Extend deployment mode support (#577) ## Changes This adds `mode: production` option. This mode doesn't do any transformations but verifies that an environment is configured correctly for production: ``` environments: prod: mode: production # paths should not be scoped to a user (unless a service principal is used) root_path: /Shared/non_user_path/... # run_as and permissions should be set at the resource level (or at the top level when that is implemented) run_as: user_name: Alice permissions: - level: CAN_MANAGE user_name: Alice ``` Additionally, this extends the existing `mode: development` option, * now prefixing deployed assets with `[dev your.user]` instead of just `[dev`] * validating that development deployments _are_ scoped to a user ## Related https://github.com/databricks/cli/pull/578/files (in draft) ## Tests Manual testing to validate the experience, error messages, and functionality with all resource types. Automated unit tests. --------- Co-authored-by: Fabian Jakobs <fabian.jakobs@databricks.com>
2023-07-30 07:19:49 +00:00
mode: production
Add job and pipeline structs (#94)
2022-11-18 10:12:24 +00:00
resources:
pipelines:
nyc_taxi_pipeline:
target: nyc_taxi_production
development: false
photon: true
Revert "Rename jobs -> workflows" (#118) This reverts PR #111. This reverts commit 230811031f54f4c9e90524a0a1863ce628f6d954.
2022-12-01 21:39:15 +00:00
jobs:
Add job and pipeline structs (#94)
2022-11-18 10:12:24 +00:00
pipeline_schedule:
name: Daily refresh of production pipeline
schedule:
quartz_cron_expression: 6 6 11 * * ?
timezone_id: UTC
tasks:
- pipeline_task:
pipeline_id: "to be interpolated"