Add a new tool changecalc that calculates all packages that need to be re-tested given
a list of changed files. Without a list, uses git to compare with main branch.
'make viewchanges' runs the tool.
'make testchanges' runs the test suite over affected packages.
New tests-changes github job is added in shadow mode that takes advantage of this tool.
## Changes
Old script could not be run from master due to security restrictions and
there is no reliable way to detect if a user as secrets.
## Tests
Opened a PR in SDK Java from fork
https://github.com/databricks/databricks-sdk-java/pull/375
## Changes
This was released 2+ months ago so it has baked enough.
Blog post: https://go.dev/blog/go1.23.
## Tests
None other than unit and integration tests.
## Changes
Automatically trigger integration tests when a PR is opened or updated
## Tests
Workflow below.
---------
Co-authored-by: Pieter Noordhuis <pieter.noordhuis@databricks.com>
## Changes
The goreleaser jobs perform a cross-platform build of the main binary
without test files. It should use a different cache than the jobs that
run tests for a single platform.
This change also updates the `release-snapshot` job to use the latest
goreleaser action, as was done in #1477.
## Tests
Ran `release-snapshot` job from this PR.
## Changes
We are starting to sign Windows CLI executables, but this has to be done
from a machine with a Yubikey storing the signing certificate for the
immediate future. As such, we will only trigger Winget publishing once
the signed binaries have been uploaded to Github.
Additionally, as an extra precaution, we will only release the signed
binaries via Winget.
## Tests
<!-- How is this tested? -->
## Changes
This PR makes changes to support creating a docker image for the CLI
with the `terraform` dependencies built in. This is useful for customers
that operate in a network-restricted environment. Normally DABs makes
API calls to registry.terraform.io to setup the terraform dependencies,
with this setup the CLI/DABs will rely on the provider binaries bundled
in the docker image.
### Specifically this PR makes the following changes:
----------------
Modifies the CLI release workflow to publish the docker images in the
Github Container Registry. URL:
https://github.com/databricks/cli/pkgs/container/cli.
We use docker support in `goreleaser` to build and publish the images.
Using goreleaser ensures the CLI packaged in the docker image is the
same release artifact as the normal releases. For more information see:
1. https://goreleaser.com/cookbooks/multi-platform-docker-images
2. https://goreleaser.com/customization/docker/
Other choices made include:
1. Using `alpine` as the base image. The reason is `alpine` is a small
and lightweight linux distribution (~5MB) and an industry standard.
2. Not using [docker
manifest](https://docs.docker.com/reference/cli/docker/manifest) to
create a multi-arch build. This is because the functionality is still
experimental.
------------------
Make the `DATABRICKS_TF_VERSION` and `DATABRICKS_TF_PROVIDER_VERSION`
environment variables optional for using the terraform file mirror.
While it's not strictly necessary to make the docker image work, it's
the "right" behaviour and reduces complexity. The rationale is:
- These environment variables here are needed so the Databricks CLI does
not accidentally use the file mirror bundled with VSCode if it's
incompatible. This does not require the env vars to be mandatory.
context: https://github.com/databricks/cli/pull/1294
- This makes the `Dockerfile` and `setup.sh` simpler. We don't need an
[entrypoint.sh script to set the version environment
variables](https://medium.com/@leonardo5621_66451/learn-how-to-use-entrypoint-scripts-in-docker-images-fede010f172d).
This also makes using an interactive terminal with `docker run -it ...`
work out of the box.
## Tests
Tested manually.
--------------------
To test the release pipeline I triggered a couple of dummy releases and
verified that the images are built successfully and uploaded to Github.
1. https://github.com/databricks/cli/pkgs/container/cli
3. workflow for release:
https://github.com/databricks/cli/actions/runs/8646106333
--------------------
I tested the docker container itself by setting up
[Charles](https://www.charlesproxy.com/) as an HTTP proxy and verifying
that no HTTP requests are made to `registry.terraform.io`
Before:
FYI, The Charles web proxy is hosted at localhost:8888.
```
shreyas.goenka@THW32HFW6T bundle-playground % rm -r .databricks
shreyas.goenka@THW32HFW6T bundle-playground % HTTP_PROXY="http://localhost:8888" HTTPS_PROXY="http://localhost:8888" cli bundle deploy
Uploading bundle files to /Users/shreyas.goenka@databricks.com/.bundle/bundle-playground/default/files...
Deploying resources...
Updating deployment state...
Deployment complete!
```
<img width="1275" alt="Screenshot 2024-04-11 at 3 21 45 PM"
src="https://github.com/databricks/cli/assets/88374338/15f37324-afbd-47c0-a40e-330ab232656b">
After:
This time bundle deploy is run from inside the docker container. We use
`host.docker.internal` to map to localhost on the host machine, and -v
to mount the host file system as a volume.
```
shreyas.goenka@THW32HFW6T bundle-playground % docker run -v ~/projects/bundle-playground:/bundle -v ~/.databrickscfg:/root/.databrickscfg -it --entrypoint /bin/sh -e HTTP_PROXY="http://host.docker.internal:8888" -e HTTPS_PROXY="http://host.docker.internal:8888" --network host ghcr.io/databricks/cli:latest-arm64
/ # cd /bundle/
/bundle # rm -r .databricks/
/bundle # databricks bundle deploy
Uploading bundle files to /Users/shreyas.goenka@databricks.com/.bundle/bundle-playground/default/files...
Deploying resources...
Updating deployment state...
Deployment complete!
```
<img width="1275" alt="Screenshot 2024-04-11 at 3 22 54 PM"
src="https://github.com/databricks/cli/assets/88374338/2a8f097e-734b-4b3e-8075-c02e98a1b275">
## Changes
This PR fixes bundle schema being broken because `for_each_task: null`
was set in the generated schema. This is not valid according to the JSON
schema specification and thus the Red Hat YAML VSCode extension was
failing to parse the YAML configuration.
This PR fixes: https://github.com/databricks/cli/issues/1312
## Tests
The fix itself was tested manually. I asserted that the autocompletion
works now. This was mistakenly overlooked the first time around when the
regression was introduced in https://github.com/databricks/cli/pull/1204
because the YAML extension provides best-effort autocomplete suggestions
even if the JSON schema fails to load.
To prevent future regressions we also add a test to assert that the JSON
schema generated itself is a valid JSON schema object. This is done via
using the `ajv-cli` to validate the schema. This package is also used by
the Red Hat YAML extension and thus provides a high fidelity check for
ensuring the JSON schema is valid.
Before, with the old schema:
```
shreyas.goenka@THW32HFW6T cli-versions % ajv validate -s proj/schema-216.json -d ../bundle-playground-3/databricks.yml
schema proj/schema-216.json is invalid
error: schema is invalid: data/properties/resources/properties/jobs/additionalProperties/properties/tasks/items/properties/for_each_task must be object,boolean, data/properties/resources/properties/jobs/additionalProperties/properties/tasks/items must be array, data/properties/resources/properties/jobs/additionalProperties/properties/tasks/items must match a schema in anyOf
```
After, with the new schema:
```
shreyas.goenka@THW32HFW6T cli-versions % ajv validate -s proj/schema-dev.json -d ../bundle-playground-3/databricks.yml
../bundle-playground-3/databricks.yml valid
```
After, autocomplete suggestions:
<img width="600" alt="Screenshot 2024-03-27 at 6 35 57 PM"
src="https://github.com/databricks/cli/assets/88374338/d0a62402-e323-4f36-854d-332b33cbeab8">
## Changes
This PR adds a release workflow which will automatically publish the CLI
to winget-pkgs whenever a release is made. It uses
https://github.com/vedantmgoyal2009/winget-releaser to release the
windows binaries. @exorcism0666 has been graciously making releases on
our behalf, but we can do this automatically ourselves after this PR.
## Tests
<!-- How is this tested? -->
## Changes
This was pinned to 1.21.0 and included a vulnerability as reported in
#1150. The vulnerability does not affect the prior CLI releases as it
requires a user to execute Go commands from within compromised module
directories.
Fixes#1150.
## Changes
This silences the following warning as seen in action output:
> Node.js 16 actions are deprecated. Please update the following actions
to use Node.js 20: actions/setup-go@v4.
## Changes
The manual unshallow step is superfluous and can be done as part of the
`actions/checkout` step.
Companion to #1022.
## Tests
Manual trigger of the snapshot build workflow.
## Changes
Version 4 enables caching by default so we no longer need to explicitly
enable it: https://github.com/actions/setup-go#v4.
The build cache only reuses a cache from a repo's default branch, which
for this repository is `main`. After enabling the merge queue, we no
longer run builds on the `main` branch after push, but on merge queue
branches. With no more builds on the `main` branch there is no longer a
cache to reuse.
This change fixes that by making the `release(-snapshot)?` workflows use
the same caching mechanism. These run off of the `main` branch, so the
cache they save can be reused by builds triggered on PRs or from the
merge queue.
## Tests
We have to merge this to see if it works.
## Changes
Commits going through the merge queue are tested there using their final
SHA as if they were already in main. The push-to-main trigger therefore
duplicates the builds that were already triggered from the merge queue.
## Tests
Denis Bilenko
Pieter Noordhuis
hectorcast-db
shreyas-goenka
Andrew Nester
Miles Yucht
Ilia Babanov