Commit Graph

111 Commits

Author SHA1 Message Date
Shreyas Goenka 44ed605a40
update main 2025-02-10 13:32:32 +01:00
Shreyas Goenka 440aafc0e5
Merge remote-tracking branch 'origin' into auth-validate 2025-02-10 13:28:55 +01:00
shreyas-goenka ee440e65fe
Serialize all header values in acceptance tests (#2311)
## Changes
Based on feedback in
https://github.com/databricks/cli/pull/2296#discussion_r1946660650.
Previously we only serialized the first value for a header in the
requests log. Now we serialise all values for a header key.

## Tests
Existing test
2025-02-10 12:18:05 +00:00
Denis Bilenko 4ebc86282f
acc: Split bundle/templates and bundle/templates-machinery (#2317)
The tests in acceptance/bundle/template focus on standard templates. The
plan is to extend them with "bundle deploy" and enable them on the
cloud.

The tests in acceptance/bundle/template-machinery focus on specific
aspects of template implementation. Most of them are expected to remain
local-only.
2025-02-10 11:55:34 +01:00
Denis Bilenko cc07380185
acc: Summarize unexpected files (#2320)
## Changes
When there are many unexpected files, it's good to see them as a list
rather than scattered throughout the output.

<!-- Summary of your changes that are easy to understand -->

## Tests
Manually, example output:
```
    acceptance_test.go:363: Test produced unexpected files:
        output/my_default_sql/.databricks/bundle/dev/sync-snapshots/71c79ded90615dc7.json
        output/my_default_sql/.databricks/bundle/dev/terraform/.terraform/providers/registry.terraform.io/databricks/databricks/1.64.1/darwin_arm64
        output/my_default_sql/.databricks/bundle/dev/terraform/plan
        output/my_default_sql/.databricks/bundle/prod/sync-snapshots/83e677e75259c93b.json
        output/my_default_sql/.databricks/bundle/prod/terraform/.terraform/providers/registry.terraform.io/databricks/databricks/1.64.1/darwin_arm64
```
2025-02-10 11:53:00 +01:00
Denis Bilenko 2175dd24a4
Do not gitignore .databricks and terraform (#2318)
For acceptance/bundle/templates I'd like to run "bundle deploy". This
would create .databricks directory inside materialized output. It might
makes sense to commit some of this as part of golden files output. Even
if we did not commit anything, the test runner will see those files and
show the difference. Thus git should also see them.

Also rename .gitignore to out.gitignore in those tests, since that
includes .databricks as well.
2025-02-10 11:42:39 +01:00
Andrew Nester f8aaa7fce3
Added support to generate Git based jobs (#2304)
## Changes
This will generate bundle YAML configuration for Git based jobs but
won't download any related files as they are in Git repo.

Fixes #1423 

## Tests
Added unit test

---------

Co-authored-by: Pieter Noordhuis <pieter.noordhuis@databricks.com>
2025-02-07 18:37:03 +00:00
Andrew Nester 2a97dcaa45
Raise an error when there are multiple local libraries with the same basename used (#2297)
## Changes
Raise an error when there are multiple local libraries with the same
basename used

Fixes #1674

## Tests
Added an unit test
2025-02-07 17:55:16 +00:00
Denis Bilenko 6d83ffd109
acc: enable bundle/scripts on cloud (#2313) 2025-02-07 17:42:47 +00:00
Denis Bilenko 989aabe5f1
acc: Make variable tests local-only (#2312)
Makes use of #2294
2025-02-07 17:42:35 +00:00
Denis Bilenko ff4a5c2269
acc: Implement config merge (#2294)
## Changes
Instead of using leaf-most config, all configs from root at
acceptance/test.toml to all intermediate ones to leaf config are merged
into one. Maps are merged, slices are appended, other values are
overridden.

I had to disable caching, because it is tricky when merging is involved
- deep copy is needed. There is performance
impact but currently it is tiny, about 1%.

Also, remove empty root config.

## Tests

Manually checked that inheritance of LocalOnly setting worked for these
tests:

Before - integration tests showed:

```
PASS acceptance.TestAccept/bundle/templates/wrong-url (0.70s)
PASS acceptance.TestAccept/bundle/templates/wrong-path (0.44s)
```

After:

```
SKIP acceptance.TestAccept/bundle/templates/wrong-url (0.00s)
SKIP acceptance.TestAccept/bundle/templates/wrong-path (0.00s)
      acceptance_test.go:216: Disabled via LocalOnly setting in bundle/templates/test.toml, bundle/templates/wrong-path/test.toml (CLOUD_ENV=***)
```
2025-02-07 17:38:27 +01:00
shreyas-goenka f71583fbc0
Error when unknown API endpoint is used in testserver (#2292)
## Changes
This PR fails the acceptance test when an unknown endpoint (i.e. not
stubbed) is used. We want to ensure that all API endpoints used in an
acceptance test are stubbed and do not otherwise silently fail with a
404.

The logs on failure output include a configuration that developers can
simply copy-paste to `test.toml` to stub the missing API endpoint. It'll
look something like:
```
[[Server]]
Pattern = "<method> <path>"
Response.Body = '''
<response body here>
'''
Response.StatusCode = <response status-code here>
```


## Tests
Manually:

output.txt when an endpoint is not found: 
```
>>> [CLI] jobs create --json {"name":"abc"}
Error: No stub found for pattern: POST /api/2.1/jobs/create
```

How this renders in the test logs:
```
    --- FAIL: TestAccept/workspace/jobs/create (0.03s)
        server.go:46: 
            
            ----------------------------------------
            No stub found for pattern: POST /api/2.1/jobs/create
            
            To stub a response for this request, you can add
            the following to test.toml:
            [[Server]]
            Pattern = "POST /api/2.1/jobs/create"
            Response.Body = '''
            <response body here>
            '''
            Response.StatusCode = <response status-code here>
            ----------------------------------------
```

Manually checked that the debug mode still works.
2025-02-07 16:26:48 +00:00
Denis Bilenko 6b1a778fe1
Fix flaky acceptance test (#2310)
## Changes
Replace timestamps with fixed string before output is sorted (and before
test runner replacements are applied).

Otherwise the test sometimes fails with error below. Note, timestamps
themselves do not show it, because they were replaced.

```
    --- FAIL: TestAccept/bundle/debug (0.78s)
        acceptance_test.go:404: Diff:
            --- bundle/debug/out.stderr.parallel.txt
            +++ /var/folders/5y/9kkdnjw91p11vsqwk0cvmk200000gp/T/TestAcceptbundledebug1859985035/001/out.stderr.parallel.txt
            @@ -8,8 +8,8 @@
             10:07:59 Debug: ApplyReadOnly pid=12345 mutator=validate mutator (read-only)=parallel mutator (read-only)=validate:files_to_sync
             10:07:59 Debug: ApplyReadOnly pid=12345 mutator=validate mutator (read-only)=parallel mutator (read-only)=validate:folder_permissions
             10:07:59 Debug: ApplyReadOnly pid=12345 mutator=validate mutator (read-only)=parallel mutator (read-only)=validate:validate_sync_patterns
            -10:07:59 Debug: Path /Workspace/Users/[USERNAME]/.bundle/debug/default/files has type directory (ID: 0) pid=12345 mutator=validate mutator (read-only)=parallel mutator (read-only)=validate:files_to_sync
             10:07:59 Debug: non-retriable error:  pid=12345 mutator=validate mutator (read-only)=parallel mutator (read-only)=validate:files_to_sync sdk=true
            +10:07:59 Debug: Path /Workspace/Users/[USERNAME]/.bundle/debug/default/files has type directory (ID: 0) pid=12345 mutator=validate mutator (read-only)=parallel mutator (read-only)=validate:files_to_sync
             < {} pid=12345 mutator=validate mutator (read-only)=parallel mutator (read-only)=validate:files_to_sync sdk=true
             < {} pid=12345 mutator=validate mutator (read-only)=parallel mutator (read-only)=validate:files_to_sync sdk=true
             < } pid=12345 mutator=validate mutator (read-only)=parallel mutator (read-only)=validate:files_to_sync sdk=true
```

## Tests
Running `hyperfine --min-runs 10 'go test ../.. -run
^TestAccept$/^bundle$/^debug$ -count=1' --show-output` detects flakiness
on main but not with this PR.
2025-02-07 16:17:50 +00:00
Denis Bilenko ecc05689ca
Add a couple of tests for bundle init with custom template (#2293)
These test custom template and what happens if helper function returns
an error.
2025-02-07 13:13:12 +00:00
Denis Bilenko 54e16d5f62
Always print warnings and errors; clean up format (#2213)
## Changes
- Print warnings and errors by default.
- Fix ErrAlreadyPrinted not to be logged at Error level.
- Format log messages as "Warn: message" instead of "WARN" to make it
more readable and in-line with the rest of the output.
- Only print attributes (pid, mutator, etc) and time when the overall
level is debug (so --debug output has not changed much).

## Tests
- Existing acceptance tests show how warning messages appear in various
test case.
- Added new test for `--debug` output.
- Add sort_lines.py helper to avoid dependency on 'sort' which is
locale-sensitive.
2025-02-07 11:29:40 +00:00
Gleb Kanterov 75127fe42e
Extend testserver for deployment (#2299)
## Changes
Extend testserver for bundle deployment:

- Allocate a new workspace per test case to isolate test cases from each
other
- Support jobs get/list/create
- Support creation and listing of workspace files

## Tests
Using existing acceptance tests
2025-02-07 10:26:20 +00:00
Shreyas Goenka 45ea71c442
- 2025-02-05 15:11:55 +01:00
Shreyas Goenka ffa8e6b01c
- 2025-02-05 15:05:49 +01:00
Shreyas Goenka 8399f0a8ca
- 2025-02-05 15:03:49 +01:00
Shreyas Goenka 61b2e22f5e
- 2025-02-05 15:03:27 +01:00
Shreyas Goenka 5a1f36e07d
- 2025-02-05 13:16:42 +01:00
Shreyas Goenka bbc23ad0f7
- 2025-02-05 13:14:10 +01:00
Shreyas Goenka 2253eb0590
merge 2025-02-05 13:13:14 +01:00
Andrew Nester 5c90752797
acc: Added acceptance test for CLI commands inside bundle with and without profile flag (#2270)
## Changes
This encodes existing behaviour in CLI as reported here: #1358
2025-02-05 11:53:36 +00:00
shreyas-goenka 57b8d336e0
Add ability to record headers in acceptance tests (#2296)
## Changes
HTTP headers like the User-Agent are an important part of our internal
ETL pipelines. This PR adds the ability to validate the headers used in
an HTTP request as part of our acceptance tests.

## Tests
Modifying existing test.
2025-02-05 09:32:15 +00:00
Simon Poltier 84b694f2a1
accept JSON includes (#2265)
#2201 disabled using JSON as part of a bundle definition. I believe this
was not intended.

## Changes
Accept json files as includes, just as YAML files.
## Tests
Covered by the tests in #2201
2025-02-04 19:28:19 +01:00
shreyas-goenka d86ad91899
Allow test servers to return errors responses (#2291)
## Changes
The APIs at Databricks when returning a non `200` status code will
return a response body of the format:
```
{
  "error_code": "Error code",
  "message": "Human-readable error message."
}
```

This PR adds the ability to stub non-200 status codes in the test
server, allowing us to mock API errors from Databricks.
## Tests
New test
2025-02-04 17:38:11 +01:00
Shreyas Goenka a40dc8cf3c
cleanup: 2025-02-03 20:05:30 +01:00
Shreyas Goenka eb8e9854d4
-gs 2025-02-03 20:01:01 +01:00
Shreyas Goenka b66bfb2c93
- 2025-02-03 19:50:16 +01:00
Shreyas Goenka b076b750f1
fix 2025-02-03 19:49:38 +01:00
Shreyas Goenka 5e9d63280a
- 2025-02-03 19:33:02 +01:00
Shreyas Goenka 9c0efb6212
- 2025-02-03 19:26:19 +01:00
Shreyas Goenka 6abfe52081
- 2025-02-03 19:25:15 +01:00
Shreyas Goenka a7e785d0e8
Add acceptance tests for auth resolution 2025-02-03 19:24:41 +01:00
Denis Bilenko 9320bd1682
acc: Use [VARNAME] instead of $VARNAME (#2282)
$VARNAME is what we use for environment variables, it's good to
separate.

Some people use envsubst for homemade variable interpolation, it's also
good to have separation there.
2025-02-03 14:10:19 +00:00
Denis Bilenko f267318bb9
Include acceptance tests in integration tests (#2242)
## Changes
- Include acceptance directory in integration tests. Acceptance tests
will not start local server if CLOUD_ENV is set, so they become
integration tests.
- Add dependency for vendor to integration, so that CLI can be build
there.
- Implement LocalOnly option in test.toml to opt out of running
acceptance tests as integration tests. Use it in certain tests that are
difficult or not necessary to fix when run as integration tests.
- Update terraform test to redact timings out.
- Clean up .workspace.current_user from outputs of the tests.

## Tests
Existing tests.
2025-02-03 10:43:25 +00:00
Denis Bilenko fcedfe4c78
acc: Consistent & detailed output for file issues (#2279)
## Changes
- Include compact relPath in the error message title. Include full paths
in separate lines below.
- Previously sometimes full paths were printed, sometime only rel path.

## Tests
Manually trigger the errors.
2025-02-03 10:29:13 +00:00
Denis Bilenko 2f798c4ded
acc: Remove initial '$CLI --version' call (#2280)
It is proven to be not necessary.

```
~/work/cli/acceptance % hyperfine -w 2 'go test'  # with change:
Benchmark 1: go test
  Time (mean ± σ):      4.983 s ±  0.209 s    [User: 6.073 s, System: 9.869 s]
  Range (min … max):    4.792 s …  5.483 s    10 runs

~/work/cli/acceptance % git stash  # without change:
~/work/cli/acceptance % hyperfine -w 2 'go test'
Benchmark 1: go test
  Time (mean ± σ):      5.018 s ±  0.100 s    [User: 6.142 s, System: 10.234 s]
  Range (min … max):    4.899 s …  5.182 s    10 runs
```
2025-02-03 10:03:18 +00:00
Denis Bilenko e5730bf57e
Use real terraform in acceptance tests (#2267)
## Changes
- Add a script install_terraform.py that downloads terraform and
provider and generates a config to use, inspired by
https://gist.github.com/pietern/1cb6b6f3e0a452328e13cdc75031105e
- Make acceptance tests run this script once before running the tests
and set the required env vars to make cli use this terraform
installation.
- Use OS-specific directory for things that are build by acceptance test
runner (CLI and terraform).

This enables acceptance tests against cloud #2242 and local test for
bundle deploy #2254.

## Tests
- Add an acceptance test for standalone terraform. This is useful to
debug terraform with TF_LOG=DEBUG to see that it uses local provider.
- Other acceptance tests are updated with regard to terraform exec path.
- The overall time for tests locally is unchanged (if terraform is
already fetched).
2025-01-31 13:53:13 +00:00
shreyas-goenka 787dbe9099
Add request body assertions to acceptance tests (#2263)
## Changes
With this PR, any acceptance tests that define custom server stubs in
`test.toml` will automatically record all HTTP requests made and assert
on them.

Builds on top of https://github.com/databricks/cli/pull/2226

## Tests
Modifying existing acceptance test.
2025-01-31 13:31:23 +00:00
shreyas-goenka 3c6eacb05b
Add feature to mock server APIs in acceptance tests (#2226)
## Changes
This PR allows us to define custom server stubs in a `test.toml` file. 

Note: A followup PR will add functionality to do assertions on the API
request itself.

## Tests
New acceptance test.
2025-01-30 10:43:07 +00:00
Denis Bilenko f1efbd7d9f
acc: add -norepl flag that disables replacements (for debugging) (#2269) 2025-01-30 10:38:54 +00:00
Denis Bilenko 58ef34f320
acc: Include "id" into /api/2.0/preview/scim/v2/Me response (#2266)
This is something terraform provider expects.

Related to https://github.com/databricks/cli/pull/2242
2025-01-29 17:35:03 +00:00
shreyas-goenka 55c03cc119
Always close test HTTP server during cleanup (#2261)
## Changes
This PR registers the `server.Close()` function to be run during test
cleanup in the server initialization function. This ensures that all
test servers are closed as soon as the test they are scoped to finish.

Motivated by https://github.com/databricks/cli/pull/2255/files where a
regression was introduced where we did not close the test server.

## Tests
N/A
2025-01-29 15:54:33 +00:00
Denis Bilenko 38efedcd73
Remove bundle.git.inferred (#2258)
The only use case for it was to emit a warning and based on the
discussion here
https://github.com/databricks/cli/pull/2213/files#r1933558087 the
warning it not useful and logging that with reduced severity is also not
useful.
2025-01-29 14:15:52 +00:00
Andrew Nester ec7808da34
Added support for double underscore variable references (#2203)
## Changes
Added support for double underscore variable references.

Previously we made this restriction stronger with no particular reason,
TF provider supports multiple underscores and thus DABs should do as
well.

Fixes #1753

## Tests
Added acceptance and integration tests
2025-01-29 13:38:28 +00:00
Ilya Kuznetsov 59d6fbfee9
Restore variable file tests (#2220)
## Changes

Uncomment flaky tests, they work properly with latest changes from main 

## Tests
<!-- How is this tested? -->
2025-01-29 13:34:26 +00:00
shreyas-goenka 124515e8d2
Move TestServer from acceptance to libs/testserver (#2255)
## Changes
Just a move, no changes. As recommended here:
https://github.com/databricks/cli/pull/2226#discussion_r1932152627

## Tests
N/A
2025-01-29 10:42:21 +00:00
Denis Bilenko 4ba222ab36
Fix env_overrides not to use variables in workspace.profile (#2251)
This does not work when this test is run against cloud.

Needed for https://github.com/databricks/cli/pull/2242
2025-01-28 15:22:56 +01:00