databricks-cli/.goreleaser.yaml

106 lines
3.4 KiB
YAML
Raw Normal View History

version: 2
2022-09-08 11:27:05 +00:00
before:
hooks:
- go mod download
builds:
- env:
- CGO_ENABLED=0
mod_timestamp: '{{ .CommitTimestamp }}'
flags:
- -trimpath
ldflags:
- '-s -w'
- -X github.com/databricks/cli/internal/build.buildProjectName={{ .ProjectName }}
- -X github.com/databricks/cli/internal/build.buildVersion={{ .Version }}
# Git information
- -X github.com/databricks/cli/internal/build.buildBranch={{ .Branch }}
- -X github.com/databricks/cli/internal/build.buildTag={{ .Tag }}
- -X github.com/databricks/cli/internal/build.buildShortCommit={{ .ShortCommit }}
- -X github.com/databricks/cli/internal/build.buildFullCommit={{ .FullCommit }}
- -X github.com/databricks/cli/internal/build.buildCommitTimestamp={{ .CommitTimestamp }}
- -X github.com/databricks/cli/internal/build.buildSummary={{ .Summary }}
# Version information
- -X github.com/databricks/cli/internal/build.buildMajor={{ .Major }}
- -X github.com/databricks/cli/internal/build.buildMinor={{ .Minor }}
- -X github.com/databricks/cli/internal/build.buildPatch={{ .Patch }}
- -X github.com/databricks/cli/internal/build.buildPrerelease={{ .Prerelease }}
- -X github.com/databricks/cli/internal/build.buildIsSnapshot={{ .IsSnapshot }}
- -X github.com/databricks/cli/internal/build.buildTimestamp={{ .Timestamp }}
goos:
- windows
- linux
- darwin
goarch:
- amd64
- arm64
binary: databricks
archives:
- format: zip
# Include version in archive only for release builds and not for snapshot builds.
# Snapshot archives must have a stable file name such that the artifacts in the nightly
# release are automatically overwritten. If the snapshot version is included in the
# file name then additional logic to clean up older builds would be needed.
name_template: 'databricks_cli_{{ if not .IsSnapshot }}{{ .Version }}_{{ end }}{{ .Os }}_{{ .Arch }}'
Add docker images for the CLI (#1353) ## Changes This PR makes changes to support creating a docker image for the CLI with the `terraform` dependencies built in. This is useful for customers that operate in a network-restricted environment. Normally DABs makes API calls to registry.terraform.io to setup the terraform dependencies, with this setup the CLI/DABs will rely on the provider binaries bundled in the docker image. ### Specifically this PR makes the following changes: ---------------- Modifies the CLI release workflow to publish the docker images in the Github Container Registry. URL: https://github.com/databricks/cli/pkgs/container/cli. We use docker support in `goreleaser` to build and publish the images. Using goreleaser ensures the CLI packaged in the docker image is the same release artifact as the normal releases. For more information see: 1. https://goreleaser.com/cookbooks/multi-platform-docker-images 2. https://goreleaser.com/customization/docker/ Other choices made include: 1. Using `alpine` as the base image. The reason is `alpine` is a small and lightweight linux distribution (~5MB) and an industry standard. 2. Not using [docker manifest](https://docs.docker.com/reference/cli/docker/manifest) to create a multi-arch build. This is because the functionality is still experimental. ------------------ Make the `DATABRICKS_TF_VERSION` and `DATABRICKS_TF_PROVIDER_VERSION` environment variables optional for using the terraform file mirror. While it's not strictly necessary to make the docker image work, it's the "right" behaviour and reduces complexity. The rationale is: - These environment variables here are needed so the Databricks CLI does not accidentally use the file mirror bundled with VSCode if it's incompatible. This does not require the env vars to be mandatory. context: https://github.com/databricks/cli/pull/1294 - This makes the `Dockerfile` and `setup.sh` simpler. We don't need an [entrypoint.sh script to set the version environment variables](https://medium.com/@leonardo5621_66451/learn-how-to-use-entrypoint-scripts-in-docker-images-fede010f172d). This also makes using an interactive terminal with `docker run -it ...` work out of the box. ## Tests Tested manually. -------------------- To test the release pipeline I triggered a couple of dummy releases and verified that the images are built successfully and uploaded to Github. 1. https://github.com/databricks/cli/pkgs/container/cli 3. workflow for release: https://github.com/databricks/cli/actions/runs/8646106333 -------------------- I tested the docker container itself by setting up [Charles](https://www.charlesproxy.com/) as an HTTP proxy and verifying that no HTTP requests are made to `registry.terraform.io` Before: FYI, The Charles web proxy is hosted at localhost:8888. ``` shreyas.goenka@THW32HFW6T bundle-playground % rm -r .databricks shreyas.goenka@THW32HFW6T bundle-playground % HTTP_PROXY="http://localhost:8888" HTTPS_PROXY="http://localhost:8888" cli bundle deploy Uploading bundle files to /Users/shreyas.goenka@databricks.com/.bundle/bundle-playground/default/files... Deploying resources... Updating deployment state... Deployment complete! ``` <img width="1275" alt="Screenshot 2024-04-11 at 3 21 45 PM" src="https://github.com/databricks/cli/assets/88374338/15f37324-afbd-47c0-a40e-330ab232656b"> After: This time bundle deploy is run from inside the docker container. We use `host.docker.internal` to map to localhost on the host machine, and -v to mount the host file system as a volume. ``` shreyas.goenka@THW32HFW6T bundle-playground % docker run -v ~/projects/bundle-playground:/bundle -v ~/.databrickscfg:/root/.databrickscfg -it --entrypoint /bin/sh -e HTTP_PROXY="http://host.docker.internal:8888" -e HTTPS_PROXY="http://host.docker.internal:8888" --network host ghcr.io/databricks/cli:latest-arm64 / # cd /bundle/ /bundle # rm -r .databricks/ /bundle # databricks bundle deploy Uploading bundle files to /Users/shreyas.goenka@databricks.com/.bundle/bundle-playground/default/files... Deploying resources... Updating deployment state... Deployment complete! ``` <img width="1275" alt="Screenshot 2024-04-11 at 3 22 54 PM" src="https://github.com/databricks/cli/assets/88374338/2a8f097e-734b-4b3e-8075-c02e98a1b275">
2024-04-12 15:22:30 +00:00
dockers:
- id: arm64
goarch: arm64
# We need to use buildx to build arm64 image on a amd64 machine.
use: buildx
image_templates:
# Docker tags can't have "+" in them, so we replace it with "-"
- 'ghcr.io/databricks/cli:{{replace .Version "+" "-"}}-arm64'
- 'ghcr.io/databricks/cli:latest-arm64'
build_flag_templates:
- "--build-arg=ARCH=arm64"
- "--platform=linux/arm64"
extra_files:
- "./docker/config.tfrc"
- "./docker/setup.sh"
- id: amd64
goarch: amd64
use: buildx
image_templates:
# Docker tags can't have "+" in them, so we replace it with "-"
- 'ghcr.io/databricks/cli:{{replace .Version "+" "-"}}-amd64'
- 'ghcr.io/databricks/cli:latest-amd64'
build_flag_templates:
- "--build-arg=ARCH=amd64"
- "--platform=linux/amd64"
extra_files:
- "./docker/config.tfrc"
- "./docker/setup.sh"
Add support for multi-arch Docker images (#1362) ## Changes This PR follows instructions in https://goreleaser.com/cookbooks/multi-platform-docker-images/ to create a multi-arch docker CLI image. Thus customers can simply specify `docker pull ghcr.io/databricks/cli:latest` to pull and run the image. The current approach uses the `docker manifest` support in goreleaser to create a multi-arch image. This has a couple of pros and cons. TLDR; The changes as is in the PR are good to go and very low risk. The information provided here is just FYI. pros: Fewer configurations/workflows for us to manage/maintain. Goreleaser makes sure the correct CLI binary is in place when building the CLI and also takes care of publishing it to the Github Container Registry. cons: Goreleaser only supports [docker manifest](https://docs.docker.com/reference/cli/docker/manifest/) to create multi-arch images. This has a few minor disadvantages: 1. `goreleaser` pushes all intermediate images (arm64 and and64 specific images) to the registry. This is required for the manifest to reference them. See: https://github.com/goreleaser/goreleaser/issues/2606 Note: We have a migration path here, if someday we stop publishing intermediate images, we can simply tag the "multi-arch" image as both `latest-amd64` and `latest-arm64`. For now, these are separate images. see: https://github.com/databricks/cli/pkgs/container/cli 2. `docker manifest` is technically an experimental command. Though it's been out for multiple years now and the indirect dependency by `goreleaser` should be fine. In any case, we can migrate by moving our docker build process off goreleaser if we need to. ## Tests Tested manually by publishing a new release for `v0.0.0-docker` in ghcr.io. 1. Package: https://github.com/databricks/cli/pkgs/container/cli 2. Release workflow: https://github.com/databricks/cli/actions/runs/8689359851 Tests the image itself by running it manually: ``` ➜ cli git:(feature/multi-arch-docker) docker pull ghcr.io/databricks/cli:latest latest: Pulling from databricks/cli bca4290a9639: Already exists 6d445556910d: Already exists Digest: sha256:82eabc500b541a89182aed4d3158c955e57c1e84d9616b76510aceb1d9024425 Status: Downloaded newer image for ghcr.io/databricks/cli:latest ghcr.io/databricks/cli:latest What's Next? View a summary of image vulnerabilities and recommendations → docker scout quickview ghcr.io/databricks/cli:latest ➜ cli git:(feature/multi-arch-docker) docker run ghcr.io/databricks/cli --version Databricks CLI v0.0.0-docker ```
2024-04-16 11:26:19 +00:00
docker_manifests:
- name_template: ghcr.io/databricks/cli:{{replace .Version "+" "-"}}
image_templates:
- ghcr.io/databricks/cli:{{replace .Version "+" "-"}}-amd64
- ghcr.io/databricks/cli:{{replace .Version "+" "-"}}-arm64
- name_template: ghcr.io/databricks/cli:latest
image_templates:
- ghcr.io/databricks/cli:latest-amd64
- ghcr.io/databricks/cli:latest-arm64
Add docker images for the CLI (#1353) ## Changes This PR makes changes to support creating a docker image for the CLI with the `terraform` dependencies built in. This is useful for customers that operate in a network-restricted environment. Normally DABs makes API calls to registry.terraform.io to setup the terraform dependencies, with this setup the CLI/DABs will rely on the provider binaries bundled in the docker image. ### Specifically this PR makes the following changes: ---------------- Modifies the CLI release workflow to publish the docker images in the Github Container Registry. URL: https://github.com/databricks/cli/pkgs/container/cli. We use docker support in `goreleaser` to build and publish the images. Using goreleaser ensures the CLI packaged in the docker image is the same release artifact as the normal releases. For more information see: 1. https://goreleaser.com/cookbooks/multi-platform-docker-images 2. https://goreleaser.com/customization/docker/ Other choices made include: 1. Using `alpine` as the base image. The reason is `alpine` is a small and lightweight linux distribution (~5MB) and an industry standard. 2. Not using [docker manifest](https://docs.docker.com/reference/cli/docker/manifest) to create a multi-arch build. This is because the functionality is still experimental. ------------------ Make the `DATABRICKS_TF_VERSION` and `DATABRICKS_TF_PROVIDER_VERSION` environment variables optional for using the terraform file mirror. While it's not strictly necessary to make the docker image work, it's the "right" behaviour and reduces complexity. The rationale is: - These environment variables here are needed so the Databricks CLI does not accidentally use the file mirror bundled with VSCode if it's incompatible. This does not require the env vars to be mandatory. context: https://github.com/databricks/cli/pull/1294 - This makes the `Dockerfile` and `setup.sh` simpler. We don't need an [entrypoint.sh script to set the version environment variables](https://medium.com/@leonardo5621_66451/learn-how-to-use-entrypoint-scripts-in-docker-images-fede010f172d). This also makes using an interactive terminal with `docker run -it ...` work out of the box. ## Tests Tested manually. -------------------- To test the release pipeline I triggered a couple of dummy releases and verified that the images are built successfully and uploaded to Github. 1. https://github.com/databricks/cli/pkgs/container/cli 3. workflow for release: https://github.com/databricks/cli/actions/runs/8646106333 -------------------- I tested the docker container itself by setting up [Charles](https://www.charlesproxy.com/) as an HTTP proxy and verifying that no HTTP requests are made to `registry.terraform.io` Before: FYI, The Charles web proxy is hosted at localhost:8888. ``` shreyas.goenka@THW32HFW6T bundle-playground % rm -r .databricks shreyas.goenka@THW32HFW6T bundle-playground % HTTP_PROXY="http://localhost:8888" HTTPS_PROXY="http://localhost:8888" cli bundle deploy Uploading bundle files to /Users/shreyas.goenka@databricks.com/.bundle/bundle-playground/default/files... Deploying resources... Updating deployment state... Deployment complete! ``` <img width="1275" alt="Screenshot 2024-04-11 at 3 21 45 PM" src="https://github.com/databricks/cli/assets/88374338/15f37324-afbd-47c0-a40e-330ab232656b"> After: This time bundle deploy is run from inside the docker container. We use `host.docker.internal` to map to localhost on the host machine, and -v to mount the host file system as a volume. ``` shreyas.goenka@THW32HFW6T bundle-playground % docker run -v ~/projects/bundle-playground:/bundle -v ~/.databrickscfg:/root/.databrickscfg -it --entrypoint /bin/sh -e HTTP_PROXY="http://host.docker.internal:8888" -e HTTPS_PROXY="http://host.docker.internal:8888" --network host ghcr.io/databricks/cli:latest-arm64 / # cd /bundle/ /bundle # rm -r .databricks/ /bundle # databricks bundle deploy Uploading bundle files to /Users/shreyas.goenka@databricks.com/.bundle/bundle-playground/default/files... Deploying resources... Updating deployment state... Deployment complete! ``` <img width="1275" alt="Screenshot 2024-04-11 at 3 22 54 PM" src="https://github.com/databricks/cli/assets/88374338/2a8f097e-734b-4b3e-8075-c02e98a1b275">
2024-04-12 15:22:30 +00:00
checksum:
name_template: 'databricks_cli_{{ .Version }}_SHA256SUMS'
algorithm: sha256
snapshot:
name_template: '{{ incpatch .Version }}-dev+{{ .ShortCommit }}'
changelog:
sort: asc
filters:
exclude:
- '^docs:'
- '^test:'