databricks-cli/.github/workflows/release.yml

Ignoring revisions in .git-blame-ignore-revs. Click here to bypass and see the normal blame view.

156 lines
4.9 KiB
YAML
Raw Normal View History

name: release
on:
push:
tags:
- "v*"
2022-09-08 12:13:43 +00:00
workflow_dispatch:
jobs:
goreleaser:
runs-on:
group: databricks-deco-testing-runner-group
labels: ubuntu-latest-deco
outputs:
artifacts: ${{ steps.releaser.outputs.artifacts }}
steps:
2022-09-08 13:12:26 +00:00
- name: Checkout repository and submodules
uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683 # v4.2.2
with:
fetch-depth: 0
fetch-tags: true
- name: Setup Go
uses: actions/setup-go@3041bf56c941b39c61721a86cd11f3bb1338122a # v5.2.0
with:
go-version-file: go.mod
# The default cache key for this action considers only the `go.sum` file.
# We include .goreleaser.yaml here to differentiate from the cache used by the push action
# that runs unit tests. This job produces and uses a different cache.
cache-dependency-path: |
go.sum
.goreleaser.yaml
Add docker images for the CLI (#1353) ## Changes This PR makes changes to support creating a docker image for the CLI with the `terraform` dependencies built in. This is useful for customers that operate in a network-restricted environment. Normally DABs makes API calls to registry.terraform.io to setup the terraform dependencies, with this setup the CLI/DABs will rely on the provider binaries bundled in the docker image. ### Specifically this PR makes the following changes: ---------------- Modifies the CLI release workflow to publish the docker images in the Github Container Registry. URL: https://github.com/databricks/cli/pkgs/container/cli. We use docker support in `goreleaser` to build and publish the images. Using goreleaser ensures the CLI packaged in the docker image is the same release artifact as the normal releases. For more information see: 1. https://goreleaser.com/cookbooks/multi-platform-docker-images 2. https://goreleaser.com/customization/docker/ Other choices made include: 1. Using `alpine` as the base image. The reason is `alpine` is a small and lightweight linux distribution (~5MB) and an industry standard. 2. Not using [docker manifest](https://docs.docker.com/reference/cli/docker/manifest) to create a multi-arch build. This is because the functionality is still experimental. ------------------ Make the `DATABRICKS_TF_VERSION` and `DATABRICKS_TF_PROVIDER_VERSION` environment variables optional for using the terraform file mirror. While it's not strictly necessary to make the docker image work, it's the "right" behaviour and reduces complexity. The rationale is: - These environment variables here are needed so the Databricks CLI does not accidentally use the file mirror bundled with VSCode if it's incompatible. This does not require the env vars to be mandatory. context: https://github.com/databricks/cli/pull/1294 - This makes the `Dockerfile` and `setup.sh` simpler. We don't need an [entrypoint.sh script to set the version environment variables](https://medium.com/@leonardo5621_66451/learn-how-to-use-entrypoint-scripts-in-docker-images-fede010f172d). This also makes using an interactive terminal with `docker run -it ...` work out of the box. ## Tests Tested manually. -------------------- To test the release pipeline I triggered a couple of dummy releases and verified that the images are built successfully and uploaded to Github. 1. https://github.com/databricks/cli/pkgs/container/cli 3. workflow for release: https://github.com/databricks/cli/actions/runs/8646106333 -------------------- I tested the docker container itself by setting up [Charles](https://www.charlesproxy.com/) as an HTTP proxy and verifying that no HTTP requests are made to `registry.terraform.io` Before: FYI, The Charles web proxy is hosted at localhost:8888. ``` shreyas.goenka@THW32HFW6T bundle-playground % rm -r .databricks shreyas.goenka@THW32HFW6T bundle-playground % HTTP_PROXY="http://localhost:8888" HTTPS_PROXY="http://localhost:8888" cli bundle deploy Uploading bundle files to /Users/shreyas.goenka@databricks.com/.bundle/bundle-playground/default/files... Deploying resources... Updating deployment state... Deployment complete! ``` <img width="1275" alt="Screenshot 2024-04-11 at 3 21 45 PM" src="https://github.com/databricks/cli/assets/88374338/15f37324-afbd-47c0-a40e-330ab232656b"> After: This time bundle deploy is run from inside the docker container. We use `host.docker.internal` to map to localhost on the host machine, and -v to mount the host file system as a volume. ``` shreyas.goenka@THW32HFW6T bundle-playground % docker run -v ~/projects/bundle-playground:/bundle -v ~/.databrickscfg:/root/.databrickscfg -it --entrypoint /bin/sh -e HTTP_PROXY="http://host.docker.internal:8888" -e HTTPS_PROXY="http://host.docker.internal:8888" --network host ghcr.io/databricks/cli:latest-arm64 / # cd /bundle/ /bundle # rm -r .databricks/ /bundle # databricks bundle deploy Uploading bundle files to /Users/shreyas.goenka@databricks.com/.bundle/bundle-playground/default/files... Deploying resources... Updating deployment state... Deployment complete! ``` <img width="1275" alt="Screenshot 2024-04-11 at 3 22 54 PM" src="https://github.com/databricks/cli/assets/88374338/2a8f097e-734b-4b3e-8075-c02e98a1b275">
2024-04-12 15:22:30 +00:00
# Log into the GitHub Container Registry. The goreleaser action will create
# the docker images and push them to the GitHub Container Registry.
- uses: docker/login-action@9780b0c442fbb1117ed29e0efdff1e18412f7567 # v3.3.0
Add docker images for the CLI (#1353) ## Changes This PR makes changes to support creating a docker image for the CLI with the `terraform` dependencies built in. This is useful for customers that operate in a network-restricted environment. Normally DABs makes API calls to registry.terraform.io to setup the terraform dependencies, with this setup the CLI/DABs will rely on the provider binaries bundled in the docker image. ### Specifically this PR makes the following changes: ---------------- Modifies the CLI release workflow to publish the docker images in the Github Container Registry. URL: https://github.com/databricks/cli/pkgs/container/cli. We use docker support in `goreleaser` to build and publish the images. Using goreleaser ensures the CLI packaged in the docker image is the same release artifact as the normal releases. For more information see: 1. https://goreleaser.com/cookbooks/multi-platform-docker-images 2. https://goreleaser.com/customization/docker/ Other choices made include: 1. Using `alpine` as the base image. The reason is `alpine` is a small and lightweight linux distribution (~5MB) and an industry standard. 2. Not using [docker manifest](https://docs.docker.com/reference/cli/docker/manifest) to create a multi-arch build. This is because the functionality is still experimental. ------------------ Make the `DATABRICKS_TF_VERSION` and `DATABRICKS_TF_PROVIDER_VERSION` environment variables optional for using the terraform file mirror. While it's not strictly necessary to make the docker image work, it's the "right" behaviour and reduces complexity. The rationale is: - These environment variables here are needed so the Databricks CLI does not accidentally use the file mirror bundled with VSCode if it's incompatible. This does not require the env vars to be mandatory. context: https://github.com/databricks/cli/pull/1294 - This makes the `Dockerfile` and `setup.sh` simpler. We don't need an [entrypoint.sh script to set the version environment variables](https://medium.com/@leonardo5621_66451/learn-how-to-use-entrypoint-scripts-in-docker-images-fede010f172d). This also makes using an interactive terminal with `docker run -it ...` work out of the box. ## Tests Tested manually. -------------------- To test the release pipeline I triggered a couple of dummy releases and verified that the images are built successfully and uploaded to Github. 1. https://github.com/databricks/cli/pkgs/container/cli 3. workflow for release: https://github.com/databricks/cli/actions/runs/8646106333 -------------------- I tested the docker container itself by setting up [Charles](https://www.charlesproxy.com/) as an HTTP proxy and verifying that no HTTP requests are made to `registry.terraform.io` Before: FYI, The Charles web proxy is hosted at localhost:8888. ``` shreyas.goenka@THW32HFW6T bundle-playground % rm -r .databricks shreyas.goenka@THW32HFW6T bundle-playground % HTTP_PROXY="http://localhost:8888" HTTPS_PROXY="http://localhost:8888" cli bundle deploy Uploading bundle files to /Users/shreyas.goenka@databricks.com/.bundle/bundle-playground/default/files... Deploying resources... Updating deployment state... Deployment complete! ``` <img width="1275" alt="Screenshot 2024-04-11 at 3 21 45 PM" src="https://github.com/databricks/cli/assets/88374338/15f37324-afbd-47c0-a40e-330ab232656b"> After: This time bundle deploy is run from inside the docker container. We use `host.docker.internal` to map to localhost on the host machine, and -v to mount the host file system as a volume. ``` shreyas.goenka@THW32HFW6T bundle-playground % docker run -v ~/projects/bundle-playground:/bundle -v ~/.databrickscfg:/root/.databrickscfg -it --entrypoint /bin/sh -e HTTP_PROXY="http://host.docker.internal:8888" -e HTTPS_PROXY="http://host.docker.internal:8888" --network host ghcr.io/databricks/cli:latest-arm64 / # cd /bundle/ /bundle # rm -r .databricks/ /bundle # databricks bundle deploy Uploading bundle files to /Users/shreyas.goenka@databricks.com/.bundle/bundle-playground/default/files... Deploying resources... Updating deployment state... Deployment complete! ``` <img width="1275" alt="Screenshot 2024-04-11 at 3 22 54 PM" src="https://github.com/databricks/cli/assets/88374338/2a8f097e-734b-4b3e-8075-c02e98a1b275">
2024-04-12 15:22:30 +00:00
with:
registry: "ghcr.io"
username: "${{ github.actor }}"
password: "${{ secrets.GITHUB_TOKEN }}"
# QEMU is required to build cross platform docker images using buildx.
# It allows virtualization of the CPU architecture at the application level.
- name: Set up QEMU dependency
build(deps): bump docker/setup-qemu-action from 3.3.0 to 3.6.0 (#2410) Bumps [docker/setup-qemu-action](https://github.com/docker/setup-qemu-action) from 3.3.0 to 3.6.0. <details> <summary>Release notes</summary> <p><em>Sourced from <a href="https://github.com/docker/setup-qemu-action/releases">docker/setup-qemu-action's releases</a>.</em></p> <blockquote> <h2>v3.6.0</h2> <ul> <li>Display binfmt version by <a href="https://github.com/crazy-max"><code>@​crazy-max</code></a> in <a href="https://redirect.github.com/docker/setup-qemu-action/pull/202">docker/setup-qemu-action#202</a></li> </ul> <p><strong>Full Changelog</strong>: <a href="https://github.com/docker/setup-qemu-action/compare/v3.5.0...v3.6.0">https://github.com/docker/setup-qemu-action/compare/v3.5.0...v3.6.0</a></p> <h2>v3.5.0</h2> <ul> <li>Bump <code>@​docker/actions-toolkit</code> from 0.54.0 to 0.56.0 in <a href="https://redirect.github.com/docker/setup-qemu-action/pull/205">docker/setup-qemu-action#205</a></li> </ul> <p><strong>Full Changelog</strong>: <a href="https://github.com/docker/setup-qemu-action/compare/v3.4.0...v3.5.0">https://github.com/docker/setup-qemu-action/compare/v3.4.0...v3.5.0</a></p> <h2>v3.4.0</h2> <ul> <li>Bump <code>@​docker/actions-toolkit</code> from 0.49.0 to 0.54.0 in <a href="https://redirect.github.com/docker/setup-qemu-action/pull/193">docker/setup-qemu-action#193</a> <a href="https://redirect.github.com/docker/setup-qemu-action/pull/197">docker/setup-qemu-action#197</a></li> </ul> <p><strong>Full Changelog</strong>: <a href="https://github.com/docker/setup-qemu-action/compare/v3.3.0...v3.4.0">https://github.com/docker/setup-qemu-action/compare/v3.3.0...v3.4.0</a></p> </blockquote> </details> <details> <summary>Commits</summary> <ul> <li><a href="https://github.com/docker/setup-qemu-action/commit/29109295f81e9208d7d86ff1c6c12d2833863392"><code>2910929</code></a> Merge pull request <a href="https://redirect.github.com/docker/setup-qemu-action/issues/202">#202</a> from crazy-max/binfmt-version</li> <li><a href="https://github.com/docker/setup-qemu-action/commit/7ffe24aa9a8440b164354dd7e2e6793b43bc73fa"><code>7ffe24a</code></a> chore: update generated content</li> <li><a href="https://github.com/docker/setup-qemu-action/commit/17bc18bb05034c78f2ede32a8bbefcb428ed3f54"><code>17bc18b</code></a> display binfmt version</li> <li><a href="https://github.com/docker/setup-qemu-action/commit/5964de0df58d5ad28b04d8fe2e6b80ad47105b91"><code>5964de0</code></a> Merge pull request <a href="https://redirect.github.com/docker/setup-qemu-action/issues/205">#205</a> from docker/dependabot/npm_and_yarn/docker/actions-to...</li> <li><a href="https://github.com/docker/setup-qemu-action/commit/862b6633f805f5f6aa43ff50a2177924d5a38852"><code>862b663</code></a> chore: update generated content</li> <li><a href="https://github.com/docker/setup-qemu-action/commit/138de3b6464b863fed6a98c1fae46faa58f98dee"><code>138de3b</code></a> build(deps): bump <code>@​docker/actions-toolkit</code> from 0.54.0 to 0.56.0</li> <li><a href="https://github.com/docker/setup-qemu-action/commit/4574d27a4764455b42196d70a065bc6853246a25"><code>4574d27</code></a> Merge pull request <a href="https://redirect.github.com/docker/setup-qemu-action/issues/195">#195</a> from radarhere/patch-1</li> <li><a href="https://github.com/docker/setup-qemu-action/commit/7a38281c3546ccdb3cedcebedb702dd0cbde7836"><code>7a38281</code></a> Merge pull request <a href="https://redirect.github.com/docker/setup-qemu-action/issues/197">#197</a> from docker/dependabot/npm_and_yarn/docker/actions-to...</li> <li><a href="https://github.com/docker/setup-qemu-action/commit/7a1c63f9e5c282d0c95da1f77892cf8c8c73d8a1"><code>7a1c63f</code></a> build(deps): bump <code>@​docker/actions-toolkit</code> from 0.53.0 to 0.54.0</li> <li><a href="https://github.com/docker/setup-qemu-action/commit/2825a1268fb8489c1191bf37f6dacdd9be57a0c6"><code>2825a12</code></a> Fixed typo</li> <li>Additional commits viewable in <a href="https://github.com/docker/setup-qemu-action/compare/53851d14592bedcffcf25ea515637cff71ef929a...29109295f81e9208d7d86ff1c6c12d2833863392">compare view</a></li> </ul> </details> <br /> [![Dependabot compatibility score](https://dependabot-badges.githubapp.com/badges/compatibility_score?dependency-name=docker/setup-qemu-action&package-manager=github_actions&previous-version=3.3.0&new-version=3.6.0)](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores) Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting `@dependabot rebase`. [//]: # (dependabot-automerge-start) [//]: # (dependabot-automerge-end) --- <details> <summary>Dependabot commands and options</summary> <br /> You can trigger Dependabot actions by commenting on this PR: - `@dependabot rebase` will rebase this PR - `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it - `@dependabot merge` will merge this PR after your CI passes on it - `@dependabot squash and merge` will squash and merge this PR after your CI passes on it - `@dependabot cancel merge` will cancel a previously requested merge and block automerging - `@dependabot reopen` will reopen this PR if it is closed - `@dependabot close` will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually - `@dependabot show <dependency name> ignore conditions` will show all of the ignore conditions of the specified dependency - `@dependabot ignore this major version` will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this minor version` will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this dependency` will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself) </details> Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2025-03-03 08:53:36 +00:00
uses: docker/setup-qemu-action@29109295f81e9208d7d86ff1c6c12d2833863392 # v3.6.0
Add docker images for the CLI (#1353) ## Changes This PR makes changes to support creating a docker image for the CLI with the `terraform` dependencies built in. This is useful for customers that operate in a network-restricted environment. Normally DABs makes API calls to registry.terraform.io to setup the terraform dependencies, with this setup the CLI/DABs will rely on the provider binaries bundled in the docker image. ### Specifically this PR makes the following changes: ---------------- Modifies the CLI release workflow to publish the docker images in the Github Container Registry. URL: https://github.com/databricks/cli/pkgs/container/cli. We use docker support in `goreleaser` to build and publish the images. Using goreleaser ensures the CLI packaged in the docker image is the same release artifact as the normal releases. For more information see: 1. https://goreleaser.com/cookbooks/multi-platform-docker-images 2. https://goreleaser.com/customization/docker/ Other choices made include: 1. Using `alpine` as the base image. The reason is `alpine` is a small and lightweight linux distribution (~5MB) and an industry standard. 2. Not using [docker manifest](https://docs.docker.com/reference/cli/docker/manifest) to create a multi-arch build. This is because the functionality is still experimental. ------------------ Make the `DATABRICKS_TF_VERSION` and `DATABRICKS_TF_PROVIDER_VERSION` environment variables optional for using the terraform file mirror. While it's not strictly necessary to make the docker image work, it's the "right" behaviour and reduces complexity. The rationale is: - These environment variables here are needed so the Databricks CLI does not accidentally use the file mirror bundled with VSCode if it's incompatible. This does not require the env vars to be mandatory. context: https://github.com/databricks/cli/pull/1294 - This makes the `Dockerfile` and `setup.sh` simpler. We don't need an [entrypoint.sh script to set the version environment variables](https://medium.com/@leonardo5621_66451/learn-how-to-use-entrypoint-scripts-in-docker-images-fede010f172d). This also makes using an interactive terminal with `docker run -it ...` work out of the box. ## Tests Tested manually. -------------------- To test the release pipeline I triggered a couple of dummy releases and verified that the images are built successfully and uploaded to Github. 1. https://github.com/databricks/cli/pkgs/container/cli 3. workflow for release: https://github.com/databricks/cli/actions/runs/8646106333 -------------------- I tested the docker container itself by setting up [Charles](https://www.charlesproxy.com/) as an HTTP proxy and verifying that no HTTP requests are made to `registry.terraform.io` Before: FYI, The Charles web proxy is hosted at localhost:8888. ``` shreyas.goenka@THW32HFW6T bundle-playground % rm -r .databricks shreyas.goenka@THW32HFW6T bundle-playground % HTTP_PROXY="http://localhost:8888" HTTPS_PROXY="http://localhost:8888" cli bundle deploy Uploading bundle files to /Users/shreyas.goenka@databricks.com/.bundle/bundle-playground/default/files... Deploying resources... Updating deployment state... Deployment complete! ``` <img width="1275" alt="Screenshot 2024-04-11 at 3 21 45 PM" src="https://github.com/databricks/cli/assets/88374338/15f37324-afbd-47c0-a40e-330ab232656b"> After: This time bundle deploy is run from inside the docker container. We use `host.docker.internal` to map to localhost on the host machine, and -v to mount the host file system as a volume. ``` shreyas.goenka@THW32HFW6T bundle-playground % docker run -v ~/projects/bundle-playground:/bundle -v ~/.databrickscfg:/root/.databrickscfg -it --entrypoint /bin/sh -e HTTP_PROXY="http://host.docker.internal:8888" -e HTTPS_PROXY="http://host.docker.internal:8888" --network host ghcr.io/databricks/cli:latest-arm64 / # cd /bundle/ /bundle # rm -r .databricks/ /bundle # databricks bundle deploy Uploading bundle files to /Users/shreyas.goenka@databricks.com/.bundle/bundle-playground/default/files... Deploying resources... Updating deployment state... Deployment complete! ``` <img width="1275" alt="Screenshot 2024-04-11 at 3 22 54 PM" src="https://github.com/databricks/cli/assets/88374338/2a8f097e-734b-4b3e-8075-c02e98a1b275">
2024-04-12 15:22:30 +00:00
- name: Run GoReleaser
id: releaser
uses: goreleaser/goreleaser-action@9ed2f89a662bf1735a48bc8557fd212fa902bebf # v6.1.0
with:
version: ~> v2
args: release
env:
2022-09-08 12:13:43 +00:00
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
create-setup-cli-release-pr:
runs-on:
group: databricks-deco-testing-runner-group
labels: ubuntu-latest-deco
needs: goreleaser
steps:
- name: Set VERSION variable from tag
run: |
VERSION=${{ github.ref_name }}
echo "VERSION=${VERSION:1}" >> $GITHUB_ENV
- name: Update setup-cli
uses: actions/github-script@60a0d83039c74a4aee543508d2ffcb1c3799cdea # v7.0.1
with:
github-token: ${{ secrets.DECO_GITHUB_TOKEN }}
script: |
await github.rest.actions.createWorkflowDispatch({
owner: 'databricks',
repo: 'setup-cli',
workflow_id: 'release-pr.yml',
ref: 'main',
inputs: {
version: "${{ env.VERSION }}",
}
});
create-homebrew-tap-release-pr:
runs-on:
group: databricks-deco-testing-runner-group
labels: ubuntu-latest-deco
needs: goreleaser
steps:
- name: Set VERSION variable from tag
run: |
VERSION=${{ github.ref_name }}
echo "VERSION=${VERSION:1}" >> $GITHUB_ENV
- name: Update homebrew-tap
uses: actions/github-script@60a0d83039c74a4aee543508d2ffcb1c3799cdea # v7.0.1
with:
github-token: ${{ secrets.DECO_GITHUB_TOKEN }}
script: |
let artifacts = ${{ needs.goreleaser.outputs.artifacts }}
artifacts = artifacts.filter(a => a.type == "Archive")
artifacts = new Map(
artifacts.map(a => [
a.goos + "_" + a.goarch,
a.extra.Checksum.replace("sha256:", "")
])
)
await github.rest.actions.createWorkflowDispatch({
owner: 'databricks',
repo: 'homebrew-tap',
workflow_id: 'release-pr.yml',
ref: 'main',
inputs: {
version: "${{ env.VERSION }}",
darwin_amd64_sha: artifacts.get('darwin_amd64'),
darwin_arm64_sha: artifacts.get('darwin_arm64'),
linux_amd64_sha: artifacts.get('linux_amd64'),
linux_arm64_sha: artifacts.get('linux_arm64')
}
});
create-vscode-extension-update-pr:
runs-on:
group: databricks-deco-testing-runner-group
labels: ubuntu-latest-deco
needs: goreleaser
steps:
- name: Set VERSION variable from tag
run: |
VERSION=${{ github.ref_name }}
echo "VERSION=${VERSION:1}" >> $GITHUB_ENV
- name: Update CLI version in the VSCode extension
uses: actions/github-script@60a0d83039c74a4aee543508d2ffcb1c3799cdea # v7.0.1
with:
github-token: ${{ secrets.DECO_GITHUB_TOKEN }}
script: |
await github.rest.actions.createWorkflowDispatch({
owner: 'databricks',
repo: 'databricks-vscode',
workflow_id: 'update-cli-version.yml',
ref: 'main',
inputs: {
version: "${{ env.VERSION }}",
}
});