Commit Graph

1513 Commits

Author SHA1 Message Date
Ilya Kuznetsov 3e0d2324b5
fix: Allow empty fields 2025-01-13 15:28:41 +01:00
Ilya Kuznetsov c88498eb56
fix: Schema update 2025-01-13 15:27:36 +01:00
Ilya Kuznetsov ad81e1f0af fix: Few links updates 2025-01-13 15:16:47 +01:00
Ilya Kuznetsov 5fb4fa000b fix: Add field-name for some properties 2025-01-10 21:01:15 +01:00
Ilya Kuznetsov fd88e4c97a fix: Use markdown from original pages 2025-01-10 20:35:44 +01:00
Ilya Kuznetsov 90cafad8c8 fix: Allow nodes with only description 2025-01-10 19:57:02 +01:00
Ilya Kuznetsov c546604cc4
fix: Schema bump 2025-01-10 19:29:43 +01:00
Ilya Kuznetsov 880a4cf49e
fix: Few extra links 2025-01-10 19:29:16 +01:00
Ilya Kuznetsov 0bd7b524db
fix: Invalid refrences 2025-01-10 19:26:45 +01:00
Ilya Kuznetsov ee5db187c0
fix: Move logic to separate fiels 2025-01-10 18:33:58 +01:00
Ilya Kuznetsov 4b01f6b1bc
fix: Add links 2025-01-10 18:09:18 +01:00
Ilya Kuznetsov 151a6f86dd
fix: Multiple links 2025-01-10 17:51:22 +01:00
Ilya Kuznetsov 1fbec371e5
fix: Link 2025-01-10 17:47:06 +01:00
Ilya Kuznetsov d5d433e9d7
fix: More descriptions 2025-01-10 16:37:22 +01:00
Ilya Kuznetsov 954ef76fe9
Merge branch 'main' of github.com:databricks/cli into feat/config-reference-doc-autogen 2025-01-10 16:07:13 +01:00
Ilya Kuznetsov bad77bd358
fix: Sync annotations 2025-01-10 16:01:52 +01:00
Ilya Kuznetsov 6c5268aaaa fix: Missing array types 2025-01-10 12:52:20 +01:00
Pieter Noordhuis dc3a157fdc
Remove cleanup in testcli package (#2108)
## Changes

The main CLI entry point used to be a global variable, and the global
state had to be cleaned up after every test run. This hasn't been the
case for a while, and instead, the CLI is initialized in a function
call. State accumulated by a single CLI "instance" can no longer leak
into other instances, so we no longer have to perform cleanup.

## Tests

Existing tests pass.
2025-01-10 10:16:53 +00:00
Denis Bilenko 99cd3fe184
Bump golangci-lint version to v1.63.4 from v1.63.1 (#2114) 2025-01-10 10:49:57 +01:00
Denis Bilenko 75cd582021
Remove lint.sh; re-add 'make fmt' (#2113)
See Makefile for explanation on difference between 'make fmt' and 'make
lint'.

I also removed lint.sh. Original motivation was to use it in aider, but
it's not a good fit there, because aider passes filenames and it does
not work well with most golang linters which requires whole packages to
work.

Follow up to #2062, #2056, #2051.
2025-01-10 10:49:33 +01:00
Denis Bilenko 72e833a897
Configure dependabot to check for new github-actions (#2112) 2025-01-10 09:39:00 +00:00
Denis Bilenko f2c4cae9f1
Increase close-after-stale from 7 to 30 days (#2111)
Giving 7 days to react before closing is too aggressive, IMO. Changed it
to 30.

Also changed 'stale' label from 30d to 60d.

Also removed dry-run setting, it does not appear to do anything.
2025-01-10 09:32:39 +00:00
Denis Bilenko 6d3b4159bd
Log warnings to stderr for "bundle validate -o json" (#2109)
## Changes
Previously diagnostics were not seen in JSON output mode. This change
prints them to stderr.

This also fixes acceptance tests to preprocess all output with
s/execPath/$CLI/ not just output.txt.

## Tests
Existing acceptance tests. In one case I've added non-json command to
check that they match in output.
2025-01-10 08:51:59 +00:00
shreyas-goenka b0c1c23630
Add `uuid` to builtin templates (#2088)
## Changes
This is useful to track telemetry associated with the templates and can
later be useful for functional usecases as well. Mlops stacks does the
same here: https://github.com/databricks/mlops-stacks/pull/185

## Tests
Existing tests.
2025-01-09 18:19:34 +00:00
Denis Bilenko a0455bcaef
Migrate bundle/tests/undefined_resources_test.go to acceptance test (#2106)
Add sort_blocks.py helper to deal with non-determinism.
2025-01-09 15:21:24 +00:00
Pieter Noordhuis 4b67e9f336
Pass tag to release as input to publish-winget workflow (#2107)
## Changes

This workflow only worked if it was triggered on the tag to publish
itself. This means it is not possible to release a version if the
workflow configuration at that tag is broken (as is the case for
v0.238.0 because of #2105).

This change adds a "tag" input that can be set when manually triggering
the workflow.

## Tests

* Succesful run with this change:
https://github.com/databricks/cli/actions/runs/12689281843
* Pull request that the run created:
https://github.com/microsoft/winget-pkgs/pull/209220
2025-01-09 12:07:29 +00:00
Pieter Noordhuis 3b3ede6e31
Update runner for the publish-winget job (#2105)
## Changes

This action uses a token to access the release artifacts and, as such,
needs to execute on the runner that's on the allowlist.

Related PRs:
* #2098
* #2077
2025-01-09 11:21:30 +00:00
Denis Bilenko 2a4fdd911e
Add a test showing bug in merge if variables are used for keys (#2101)
Since merge happens first, before variable resolution, the two jobs are
seen as different.

I also updated override/job_cluster/script to include more of the
output.
2025-01-09 10:14:47 +00:00
Denis Bilenko b808d4d6f1
Add test for overriding list variable (#2099)
- Add override for "libraries".
- Remove complexvar - it serves no purpose - we already have map
variable ‘cluster’.
2025-01-09 09:03:03 +00:00
Denis Bilenko b0706ccdc1
Use -update instead of TESTS_OUTPUT=OVERWRITE (#2097)
It's easier to remember and type and also validated and part of help:

```
  % go test ./acceptance -updat 2>&1 | grep updat
  flag provided but not defined: -updat
    -update
```
2025-01-09 09:00:05 +00:00
Denis Bilenko df17e4b4ea
Convert some resolve variables tests to acceptance test (#2100) 2025-01-08 17:44:52 +00:00
Pieter Noordhuis 42b34c7bef
Update runner for the create-xyz-pr jobs in the release workflow (#2098)
## Changes

I missed these in #2077 and they failed because of it on the v0.238.0
release.
2025-01-08 15:50:44 +00:00
Ilya Kuznetsov f9278c2b4d docs: Add override for volume spec 2025-01-08 16:11:58 +01:00
Ilya Kuznetsov c355fbfcb2 fix: Description of root types with additional properties 2025-01-08 16:11:35 +01:00
Pieter Noordhuis b48fa70984
[Release] Release v0.238.0 (#2096)
Bundles:
* Fix finding Python within virtualenv on Windows
([#2034](https://github.com/databricks/cli/pull/2034)).
* Include missing field descriptions in JSON schema
([#2045](https://github.com/databricks/cli/pull/2045)).
* Add validation for volume referenced from `artifact_path`
([#2050](https://github.com/databricks/cli/pull/2050)).
* Handle `${workspace.file_path}` references in source-linked
deployments ([#2046](https://github.com/databricks/cli/pull/2046)).
* Set the write bit for files written during template initialization
([#2068](https://github.com/databricks/cli/pull/2068)).
2025-01-08 14:47:38 +00:00
Pieter Noordhuis 23f05f5d67
Set the write bit for files written during template initialization (#2068)
## Changes

This used to work because the permission bits for built-in templates
were hardcoded to 0644 for files and 0755 for directories.

As of #1912 (and the PRs it depends on), built-in templates are no
longer pre-materialized to a temporary directory and read directly from
the embedded filesystem. This built-in filesystem returns 0444 as the
permission bits for the files it contains. These bits are carried over
to the destination filesystem.

This change updates template materialization to always set the owner's
write bit. It doesn't really make sense to write read-only files and
expect users to work with these files in a VCS (note: Git only stores
the executable bit).

The regression shipped as part of v0.235.0 and will be fixed as of
v0.238.0.

## Tests

Unit tests.
2025-01-08 13:18:28 +00:00
Ilya Kuznetsov 0289becea8
Handle `${workspace.file_path}` references in source-linked deployments (#2046)
## Changes

1. Updates `workspace.file_path` during source-linked deployment to
address cases like this
https://github.com/databricks/bundle-examples/blob/main/default_python/resources/default_python_pipeline.yml#L13
2. Updates `workspace.file_path` in `metadata.json`
3. Prints warning for users when `workspace.file_path` is explicitly set
but deploy is running in source-linked mode

## Tests

Unit test
2025-01-08 12:43:56 +00:00
Denis Bilenko 185bbd28e4
Add acceptance tests (#2081)
## Changes
- New kind of test is added - acceptance tests. See acceptance/README.md
for explanation.
- A few tests are converted to acceptance tests by moving databricks.yml
to acceptance/ and adding corresponding script files.

As these tests run against compiled binary and can capture full output
of the command, they can be useful to support major changes such as
refactoring internal logging / diagnostics or complex variable
interpolation.

These are currently run as part of 'make test' but the intention is to
run them as part of integration tests as well.

### Benefits

- Full binary is tested, exactly as users get it.
  - We're not testing custom set of mutators like many existing tests.
- Not mocking anything, real SDK is used (although the HTTP endpoint is
not a real Databricks env).
- Easy to maintain: output can be updated automatically.
- Can easily set up external env, such as env vars, CLI args,
.databrickscfg location etc.

### Gaps

The tests currently share the test server and there is global place to
define handlers. We should have a way for tests to override / add new
handlers.

## Tests
I manually checked that output of new acceptance tests matches previous
asserts.
2025-01-08 12:41:08 +00:00
Denis Bilenko 8fd793b605
Clean up TestMain from integration tests to fix caching (#2090)
## Changes
- Remove TestMain from integration tests and related checks.
- This fixes "go test" caching for integration tests.

The test_main.go files were added in
https://github.com/databricks/cli/pull/2009 to make sure integration
tests are not run as part of go test ./.... We recommend running make
test to run tests, which includes the packages to test (and excludes
integration).

## Tests
To test that caching works I ran a test twice:

```
+ CLOUD_ENV=aws
+ go test --timeout 3h -v -run TestDefaultPython/3.9 ./integration/bundle/
…
PASS
ok      github.com/databricks/cli/integration/bundle    (cached)
```
2025-01-08 11:59:22 +00:00
Gleb Kanterov 02c7df39f6
Add 'experimental/python' support (#2052)
## Changes
Add `experimental/python` section replacing `experimental/pydabs`.

Add 2 new mutators into existing pipeline:
- `ApplyPythonMutator(load_resources)` - loads resources from Python
code
- `ApplyPythonMutator(apply_mutators)` - transforms existing resources
defined in Python/YAML

Example:
```yaml
experimental:
  python:
    resources:
    - "resources:load_resources"
    mutators:
    - "mutators:add_email_notifications"
```

## Tests
Unit tests and manually

---------

Co-authored-by: Pieter Noordhuis <pieter.noordhuis@databricks.com>
2025-01-08 09:29:45 +00:00
Denis Bilenko 43420d01ad
Run push.yml periodically with clean cache (#2092)
This ensures that our build still works with clean cache and also
populates build/test cache, speeding up test runs.
2025-01-07 18:10:49 +01:00
Denis Bilenko e6552231eb
Use different cache keys for different jobs (#2091)
Otherwise all those jobs compete for the same key and build cache ends
up disabled for ubuntu tests.
2025-01-07 17:13:12 +01:00
Ilya Kuznetsov fe6ba76b7d fix: Styling 2025-01-07 16:52:01 +01:00
Denis Bilenko e2cd8c2f34
Enable perfsprint linter and apply autofix (#2071)
https://github.com/catenacyber/perfsprint
2025-01-07 10:49:23 +00:00
Pieter Noordhuis 3629c9e406
Enable integration workflows (#2086)
## Changes

This reverts commit 31552852ff.

These workflows were disabled in #2085.

They should work again now that we're using self-hosted runners (see
#2077).

## Tests

(inline)
2025-01-06 16:07:15 +00:00
Pieter Noordhuis c262b30ef4
Migrate workflows that need write access to use hosted runners (#2077)
## Changes

Migrate workflows to Databricks-hosted GitHub Actions runners.

The GitHub-hosted runners can no longer be used because of security
hardening.
2025-01-06 15:34:42 +00:00
Pieter Noordhuis 31552852ff
Disable integration workflows (#2085)
See https://github.com/databricks/cli/issues/2084.
2025-01-06 14:30:48 +00:00
Denis Bilenko 8af98accee
Set gotestsum --format to github-actions when running on github (#2082)
This gives grouping and ability to open individual test logs.
2025-01-06 14:12:34 +01:00
Pieter Noordhuis 4f3cf6ac2c
Limit comment about integration tests to PRs from forks (#2075)
## Changes

The comment block appears on all PRs, even if the integration tests are
automatically triggered. This is quite noisy. This change limits those
comments to PRs from forks.

## Tests

Have to try by merging...
2025-01-03 14:12:55 +00:00
Ilya Kuznetsov c6703c1363 fix: Updated styles 2025-01-03 13:52:59 +01:00