For acceptance/bundle/templates I'd like to run "bundle deploy". This would create .databricks directory inside materialized output. It might makes sense to commit some of this as part of golden files output. Even if we did not commit anything, the test runner will see those files and show the difference. Thus git should also see them. Also rename .gitignore to out.gitignore in those tests, since that includes .databricks as well. |
||
---|---|---|
.. | ||
.vscode | ||
fixtures | ||
resources | ||
scratch | ||
src | ||
tests | ||
README.md | ||
databricks.yml | ||
out.gitignore | ||
pytest.ini | ||
requirements-dev.txt | ||
setup.py |
README.md
my_default_python
The 'my_default_python' project was generated by using the default-python template.
Getting started
-
Install the Databricks CLI from https://docs.databricks.com/dev-tools/cli/databricks-cli.html
-
Authenticate to your Databricks workspace, if you have not done so already:
$ databricks configure
-
To deploy a development copy of this project, type:
$ databricks bundle deploy --target dev
(Note that "dev" is the default target, so the
--target
parameter is optional here.)This deploys everything that's defined for this project. For example, the default template would deploy a job called
[dev yourname] my_default_python_job
to your workspace. You can find that job by opening your workpace and clicking on Workflows. -
Similarly, to deploy a production copy, type:
$ databricks bundle deploy --target prod
Note that the default job from the template has a schedule that runs every day (defined in resources/my_default_python.job.yml). The schedule is paused when deploying in development mode (see https://docs.databricks.com/dev-tools/bundles/deployment-modes.html).
-
To run a job or pipeline, use the "run" command:
$ databricks bundle run
-
Optionally, install developer tools such as the Databricks extension for Visual Studio Code from https://docs.databricks.com/dev-tools/vscode-ext.html. Or read the "getting started" documentation for Databricks Connect for instructions on running the included Python code from a different IDE.
-
For documentation on the Databricks asset bundles format used for this project, and for CI/CD configuration, see https://docs.databricks.com/dev-tools/bundles/index.html.