Commit Graph

88 Commits

Author SHA1 Message Date
Ilya Kuznetsov 34d7484b75
fix: Update test to use dynamic values for comparison instead of raw file content 2024-12-17 16:43:27 +01:00
Ilya Kuznetsov 7d8ae48f6b fix: Use OS-aware path joiner 2024-12-17 15:07:22 +01:00
Ilya Kuznetsov 00376711a3 fix: More explicit names and descriptions for `annotationHandler.sync` method 2024-12-17 15:04:19 +01:00
Ilya Kuznetsov 304918436e test: Test cases for markdown link converter 2024-12-17 14:59:47 +01:00
Ilya Kuznetsov c619a73232 fix: More explicit names for annotation data structures 2024-12-17 14:55:57 +01:00
Ilya Kuznetsov 3231cffc54 fix: Remove unnecessary `ok` check 2024-12-17 14:51:09 +01:00
Ilya Kuznetsov 9f3ec3d0b4 fix: Genkit re-generate 2024-12-17 14:50:48 +01:00
Ilya Kuznetsov 7fb1ed8be6 fix: Typo in assignAnnotation 2024-12-17 14:17:13 +01:00
Ilya Kuznetsov 1b31c098bf fix: Example structure in annotationFile comment 2024-12-17 14:16:54 +01:00
Pieter Noordhuis 6bc9664399
Run make lint 2024-12-17 11:48:09 +01:00
Pieter Noordhuis 8a33fb3e95
Merge branch 'main' into feat/custom-annotations-json-schema 2024-12-17 11:47:24 +01:00
dependabot[bot] d929ea3eef
Bump golang.org/x/crypto from 0.30.0 to 0.31.0 in /bundle/internal/tf/codegen (#2005)
Bumps [golang.org/x/crypto](https://github.com/golang/crypto) from
0.30.0 to 0.31.0.
<details>
<summary>Commits</summary>
<ul>
<li><a
href="b4f1988a35"><code>b4f1988</code></a>
ssh: make the public key cache a 1-entry FIFO cache</li>
<li>See full diff in <a
href="https://github.com/golang/crypto/compare/v0.30.0...v0.31.0">compare
view</a></li>
</ul>
</details>
<br />


[![Dependabot compatibility
score](https://dependabot-badges.githubapp.com/badges/compatibility_score?dependency-name=golang.org/x/crypto&package-manager=go_modules&previous-version=0.30.0&new-version=0.31.0)](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores)

Dependabot will resolve any conflicts with this PR as long as you don't
alter it yourself. You can also trigger a rebase manually by commenting
`@dependabot rebase`.

[//]: # (dependabot-automerge-start)
[//]: # (dependabot-automerge-end)

---

<details>
<summary>Dependabot commands and options</summary>
<br />

You can trigger Dependabot actions by commenting on this PR:
- `@dependabot rebase` will rebase this PR
- `@dependabot recreate` will recreate this PR, overwriting any edits
that have been made to it
- `@dependabot merge` will merge this PR after your CI passes on it
- `@dependabot squash and merge` will squash and merge this PR after
your CI passes on it
- `@dependabot cancel merge` will cancel a previously requested merge
and block automerging
- `@dependabot reopen` will reopen this PR if it is closed
- `@dependabot close` will close this PR and stop Dependabot recreating
it. You can achieve the same result by closing it manually
- `@dependabot show <dependency name> ignore conditions` will show all
of the ignore conditions of the specified dependency
- `@dependabot ignore this major version` will close this PR and stop
Dependabot creating any more for this major version (unless you reopen
the PR or upgrade to it yourself)
- `@dependabot ignore this minor version` will close this PR and stop
Dependabot creating any more for this minor version (unless you reopen
the PR or upgrade to it yourself)
- `@dependabot ignore this dependency` will close this PR and stop
Dependabot creating any more for this dependency (unless you reopen the
PR or upgrade to it yourself)
You can disable automated security fix PRs for this repo from the
[Security Alerts
page](https://github.com/databricks/cli/network/alerts).

</details>

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2024-12-16 08:55:33 +01:00
Andrew Nester 58dfa70e50
Upgrade TF provider to 1.61.0 (#2011)
## Changes
Upgraded to TF provider 1.61.0

### New Features and Improvements
- Added databricks_app resource and data source
(https://github.com/databricks/terraform-provider-databricks/pull/4099).
- Added databricks_credentials resource and data source
2024-12-13 15:55:49 +00:00
Ilya Kuznetsov 5194889605
fix: Cleanup 2024-12-12 13:05:55 +01:00
Ilya Kuznetsov aed0e0a92f
fix: Missing annotations 2024-12-12 13:01:49 +01:00
Ilya Kuznetsov 43c4b581d4
feat: Use OneOf in interpolation patterns 2024-12-12 12:44:36 +01:00
Denis Bilenko 2e018cfaec
Enable gofumpt and goimports in golangci-lint (#1999)
## Changes
Enable gofumpt and goimports in golangci-lint and apply autofix.

This makes 'make fmt' redundant, will be cleaned up in follow up diff.

## Tests
Existing tests.
2024-12-12 10:28:42 +01:00
Ilya Kuznetsov 40505b867a
fix: Annotations cleanup 2024-12-11 14:31:53 +01:00
Ilya Kuznetsov d2bfb67be2
fix: Few descriptions fixes 2024-12-10 16:55:07 +01:00
Ilya Kuznetsov 9a5503755a
feat: Format markdown links to absolute 2024-12-10 16:13:51 +01:00
Ilya Kuznetsov e15107fbcc
feat: New annotations 2024-12-10 16:07:25 +01:00
Ilya Kuznetsov a579b1cd03
fix: Use `convert.FromTyped` to generate dyn.Value 2024-12-10 15:51:13 +01:00
Ilya Kuznetsov 00164a8dbf
fix: Linter fix for missing error handler 2024-12-10 14:54:31 +01:00
Ilya Kuznetsov d1649682e6
Merge branch 'main' of github.com:databricks/cli into feat/custom-annotations-json-schema 2024-12-10 14:40:53 +01:00
Ilya Kuznetsov a6c45d5bf7
feat: Yaml styles for open api overrides 2024-12-10 14:00:00 +01:00
Ilya Kuznetsov 073aeca5b7
feat: Add styles 2024-12-10 13:44:20 +01:00
Ilya Kuznetsov b076813e38
fix: Change the structure of annotation files, minor fixes and tests 2024-12-10 12:54:50 +01:00
Pieter Noordhuis 48d7c08a46
Bump TF codegen dependencies to latest (#1961)
## Changes

This updates the TF codegen dependencies to latest.

## Tests

Ran codegen and confirmed it still works.

See `bundle/internal/tf/codegen/README.md` for instructions.
2024-12-09 21:29:11 +01:00
Ilya Kuznetsov c3d049f3e5
fix: Custom typePath function instead of exposing private `jsonschema.typePath` function 2024-12-09 16:23:02 +01:00
Denis Bilenko 1b2be1b2cb
Add error checking in tests and enable errcheck there (#1980)
## Changes
Fix all errcheck-found issues in tests and test helpers. Mostly this
done by adding require.NoError(t, err), sometimes panic() where t object
is not available).

Initial change is obtained with aider+claude, then manually reviewed and
cleaned up.

## Tests
Existing tests.
2024-12-09 13:56:41 +01:00
Ilya Kuznetsov a3fdc8dd6c
fix: Skip tests on windows 2024-12-06 19:31:19 +01:00
Ilya Kuznetsov 0221c28a80
fix: Remove redundant field 2024-12-06 19:23:14 +01:00
Ilya Kuznetsov 8016d41661
fix: Linter fix 2024-12-06 19:22:29 +01:00
Ilya Kuznetsov 049e11b139
test: Checks missing descriptions in annotation files 2024-12-06 16:13:55 +01:00
Ilya Kuznetsov 12c43737a8
feat: Introduce separate files for OpenAPI annotations 2024-12-06 15:07:05 +01:00
Pieter Noordhuis 6e754d4f34
Rewrite 'interface{} -> any' (#1959)
## Changes

The `any` alias for `interface{}` has been around since Go 1.18.

Now that we're using golangci-lint (#1953), we can lint on it.

Existing commits can be updated with:
```
gofmt -w -r 'interface{} -> any' .
```

## Tests

n/a
2024-12-05 15:37:24 +00:00
Pieter Noordhuis 647b09e6e2
Upgrade TF provider to 1.59.0 (#1960)
## Changes

Notable changes:
* Fixes dashboard deployment if it was trashed out-of-band.
* Removes client-side validation for single-node cluster configuration
(also see #1546).

Beware: for the same reason as in #1900, this excludes the changes for
the quality monitor resource.

## Tests

Integration tests pass.
2024-12-05 12:09:45 +01:00
Ilya Kuznetsov 9b588c0ddf feat: Custom annotations for json-schema fields in bundle 2024-12-04 22:50:34 +01:00
shreyas-goenka 0da17f6ec6
Add default value for `volume_type` for DABs (#1952)
## Changes

The Unity Catalog volumes API requires a `volume_type` argument when
creating volumes. In the context of DABs, it's unnecessary to require
users to specify the volume type every time. We can default to "MANAGED"
instead.

This PR is similar to https://github.com/databricks/cli/pull/1743 which
does the same for dashboards.

## Tests
Unit test
2024-12-04 11:05:54 +00:00
Pieter Noordhuis 1db384018c
Make `TableName` field part of quality monitor schema (#1903)
## Changes

This field was special-cased in #1307 because it's not part of the JSON
payload in the SDK struct.

This approach, while pragmatic, meant it didn't show up in the JSON
schema. While debugging an issue with quality monitors in #1900, I
couldn't figure out why I was getting schema errors on this field, or
how it was passed through to the TF representation. This commit removes
the special case and makes it behave like everything else.

## Tests

* Unit tests pass.
* Confirmed that the updated schema failed validation before this
change.
2024-11-14 17:39:38 +00:00
Pieter Noordhuis 21d27885dc
Upgrade TF provider to 1.58.0 (#1900)
## Changes

Notable changes:
* Adds support for `restart_window` for pipelines.
* Fix drift for pipelines where `catalog` contains uppercase characters.
* Better error message if single-node job clusters are incorrectly configured.

See:
* https://github.com/databricks/terraform-provider-databricks/releases/tag/v1.58.0
* https://github.com/databricks/terraform-provider-databricks/releases/tag/v1.57.0
* https://github.com/databricks/terraform-provider-databricks/releases/tag/v1.56.0
* https://github.com/databricks/terraform-provider-databricks/releases/tag/v1.55.0

## Tests

Integration tests pass.
2024-11-14 14:00:15 +01:00
Pieter Noordhuis 11f75fd320
Add support for AI/BI dashboards (#1743)
## Changes

This change adds support for modeling [AI/BI dashboards][docs] in DABs.


[Example bundle configuration][example] is located in the
`bundle-examples` repository.

[docs]: https://docs.databricks.com/en/dashboards/index.html#dashboards
[example]:
https://github.com/databricks/bundle-examples/tree/main/knowledge_base/dashboard_nyc_taxi

## Tests

* Added unit tests for self-contained parts
* Integration test for e2e dashboard deployment and remote change
modification
2024-10-29 09:11:08 +00:00
Andrew Nester 68d69d6e0b
Upgrade TF provider to 1.54.0 (#1852)
## Changes
Upgrade TF provider to 1.54.0
2024-10-22 10:43:43 +00:00
Pieter Noordhuis ae568743c5
Upgrade TF provider to 1.53.0 (#1815)
## Changes

See
https://github.com/databricks/terraform-provider-databricks/releases/tag/v1.53.0

## Tests

Integration tests pass.
2024-10-08 11:41:32 +00:00
shreyas-goenka 88318d384a
Remove deprecated or readonly fields from the bundle schema (#1809)
## Changes
We do not need the user to specify these fields in their bundle
configuration, so we remove them from the JSON schema.

## Tests
Manually and end to end tests. The JSON schema has also been regenerated
after these changes.
2024-10-07 15:13:42 +00:00
shreyas-goenka 495040e4cd
Modify SetLocation test utility to take full locations as argument (#1788)
I plan to use this in https://github.com/databricks/cli/pull/1780, to
set the line and column numbers as well for the locations.

gopatch file used:
```
@@
var x expression
var y expression
var z expression
@@
-bundletest.SetLocation(x, y, z)
+bundletest.SetLocation(x, y, []dyn.Location{{File: z}})
```
2024-09-25 16:13:48 +00:00
Andrew Nester cf989a7e10
Upgrade to TF provider 1.52 (#1781)
## Changes
Upgrade to TF provider 1.52

We also temporarily skip generating plugin framework structs to unblock
upgrade as generation does not work yet and need to be fixed separately
2024-09-19 11:21:32 +00:00
shreyas-goenka c61358407f
Add end to end integration tests for bundle JSON schema (#1726) 2024-09-11 09:15:56 +00:00
shreyas-goenka 5d2c0e3885
Alias variables block in the `Target` struct (#1748)
## Changes
This PR aliases and overrides the schema associated with the variables
block in `target` to allow for directly specifying a variable value in
the JSON schema (without an levels of nesting). This is needed because
this direct value is resolved by dynamically parsing the configuration
tree.

ca6332a5a4/bundle/config/root.go (L424)

## Tests
Existing unit tests.
2024-09-10 14:49:34 +00:00
shreyas-goenka 28b39cd3f7
Make bundle JSON schema modular with `$defs` (#1700)
## Changes
This PR makes sweeping changes to the way we generate and test the
bundle JSON schema. The main benefits are:

1. More modular JSON schema. Every definition in the schema now is one
level deep and points to references instead of inlining the entire
schema for a field. This unblocks PyDABs from taking a dependency on the
JSON schema.

2. Generate the JSON schema during CLI code generation. Directly stream
it instead of computing it at runtime whenever a user calls `databricks
bundle schema`. This is nice because we no longer need to embed a
partial OpenAPI spec in the CLI. Down the line, we can add a `Schema()`
method to every struct in the Databricks Go SDK and remove the
dependency on the OpenAPI spec altogether. It'll become more important
once we decouple Go SDK structs and methods from the underlying APIs.

3. Add enum values for Go SDK fields in the JSON schema. Better
autocompletion and validation for these fields. As a follow-up, we can
add enum values for non-Go SDK enums as well (created internal ticket to
track).

4. Use "packageName.structName" as a key to read JSON schemas from the
OpenAPI spec for Go SDK structs. Before, we would use an unrolled
presentation of the JSON schema (stored in `bundle_descriptions.json`),
which was complex to parse and include in the final JSON schema output.
This also means loading values from the OpenAPI spec for `target` schema
works automatically and no longer needs custom code.
5. Support recursive types (eg: `for_each_task`). With us now using
$refs everywhere it's trivial to support.
6. Using complex variables would be invalid according to the schema
generated before this PR. Now that bug is fixed. In the future adding
more custom rules will be easier as well due to the single level nature
of the JSON schema.


Since this is a complete change of approach in how we generate the JSON
schema, there are a few (very minor) regressions worth calling out.
1. We'll lose a few custom descriptions for non Go SDK structs that were
a part of `bundle_descriptions.json`. Support for those can be added in
the future as a followup.
2. Since now the final JSON schema is a static artefact, we lose some
lead time for the signal that JSON schema integration tests are failing.
It's okay though since we have a lot of coverage via the existing unit
tests.

## Tests
Unit tests. End to end tests are being added in this PR:
https://github.com/databricks/cli/pull/1726

Previous unit tests were all deleted because they were bloated. Effort
was made to make the new unit tests provide (almost) equivalent
coverage.
2024-09-10 13:55:18 +00:00