The tests in acceptance/bundle/template focus on standard templates. The
plan is to extend them with "bundle deploy" and enable them on the
cloud.
The tests in acceptance/bundle/template-machinery focus on specific
aspects of template implementation. Most of them are expected to remain
local-only.
## Changes
When there are many unexpected files, it's good to see them as a list
rather than scattered throughout the output.
<!-- Summary of your changes that are easy to understand -->
## Tests
Manually, example output:
```
acceptance_test.go:363: Test produced unexpected files:
output/my_default_sql/.databricks/bundle/dev/sync-snapshots/71c79ded90615dc7.json
output/my_default_sql/.databricks/bundle/dev/terraform/.terraform/providers/registry.terraform.io/databricks/databricks/1.64.1/darwin_arm64
output/my_default_sql/.databricks/bundle/dev/terraform/plan
output/my_default_sql/.databricks/bundle/prod/sync-snapshots/83e677e75259c93b.json
output/my_default_sql/.databricks/bundle/prod/terraform/.terraform/providers/registry.terraform.io/databricks/databricks/1.64.1/darwin_arm64
```
For acceptance/bundle/templates I'd like to run "bundle deploy". This
would create .databricks directory inside materialized output. It might
makes sense to commit some of this as part of golden files output. Even
if we did not commit anything, the test runner will see those files and
show the difference. Thus git should also see them.
Also rename .gitignore to out.gitignore in those tests, since that
includes .databricks as well.
## Changes
This will generate bundle YAML configuration for Git based jobs but
won't download any related files as they are in Git repo.
Fixes#1423
## Tests
Added unit test
---------
Co-authored-by: Pieter Noordhuis <pieter.noordhuis@databricks.com>
## Changes
Instead of using leaf-most config, all configs from root at
acceptance/test.toml to all intermediate ones to leaf config are merged
into one. Maps are merged, slices are appended, other values are
overridden.
I had to disable caching, because it is tricky when merging is involved
- deep copy is needed. There is performance
impact but currently it is tiny, about 1%.
Also, remove empty root config.
## Tests
Manually checked that inheritance of LocalOnly setting worked for these
tests:
Before - integration tests showed:
```
PASS acceptance.TestAccept/bundle/templates/wrong-url (0.70s)
PASS acceptance.TestAccept/bundle/templates/wrong-path (0.44s)
```
After:
```
SKIP acceptance.TestAccept/bundle/templates/wrong-url (0.00s)
SKIP acceptance.TestAccept/bundle/templates/wrong-path (0.00s)
acceptance_test.go:216: Disabled via LocalOnly setting in bundle/templates/test.toml, bundle/templates/wrong-path/test.toml (CLOUD_ENV=***)
```
## Changes
This PR fails the acceptance test when an unknown endpoint (i.e. not
stubbed) is used. We want to ensure that all API endpoints used in an
acceptance test are stubbed and do not otherwise silently fail with a
404.
The logs on failure output include a configuration that developers can
simply copy-paste to `test.toml` to stub the missing API endpoint. It'll
look something like:
```
[[Server]]
Pattern = "<method> <path>"
Response.Body = '''
<response body here>
'''
Response.StatusCode = <response status-code here>
```
## Tests
Manually:
output.txt when an endpoint is not found:
```
>>> [CLI] jobs create --json {"name":"abc"}
Error: No stub found for pattern: POST /api/2.1/jobs/create
```
How this renders in the test logs:
```
--- FAIL: TestAccept/workspace/jobs/create (0.03s)
server.go:46:
----------------------------------------
No stub found for pattern: POST /api/2.1/jobs/create
To stub a response for this request, you can add
the following to test.toml:
[[Server]]
Pattern = "POST /api/2.1/jobs/create"
Response.Body = '''
<response body here>
'''
Response.StatusCode = <response status-code here>
----------------------------------------
```
Manually checked that the debug mode still works.
## Changes
- Print warnings and errors by default.
- Fix ErrAlreadyPrinted not to be logged at Error level.
- Format log messages as "Warn: message" instead of "WARN" to make it
more readable and in-line with the rest of the output.
- Only print attributes (pid, mutator, etc) and time when the overall
level is debug (so --debug output has not changed much).
## Tests
- Existing acceptance tests show how warning messages appear in various
test case.
- Added new test for `--debug` output.
- Add sort_lines.py helper to avoid dependency on 'sort' which is
locale-sensitive.
## Changes
Extend testserver for bundle deployment:
- Allocate a new workspace per test case to isolate test cases from each
other
- Support jobs get/list/create
- Support creation and listing of workspace files
## Tests
Using existing acceptance tests
## Changes
HTTP headers like the User-Agent are an important part of our internal
ETL pipelines. This PR adds the ability to validate the headers used in
an HTTP request as part of our acceptance tests.
## Tests
Modifying existing test.
#2201 disabled using JSON as part of a bundle definition. I believe this
was not intended.
## Changes
Accept json files as includes, just as YAML files.
## Tests
Covered by the tests in #2201
## Changes
The APIs at Databricks when returning a non `200` status code will
return a response body of the format:
```
{
"error_code": "Error code",
"message": "Human-readable error message."
}
```
This PR adds the ability to stub non-200 status codes in the test
server, allowing us to mock API errors from Databricks.
## Tests
New test
$VARNAME is what we use for environment variables, it's good to
separate.
Some people use envsubst for homemade variable interpolation, it's also
good to have separation there.
## Changes
- Include acceptance directory in integration tests. Acceptance tests
will not start local server if CLOUD_ENV is set, so they become
integration tests.
- Add dependency for vendor to integration, so that CLI can be build
there.
- Implement LocalOnly option in test.toml to opt out of running
acceptance tests as integration tests. Use it in certain tests that are
difficult or not necessary to fix when run as integration tests.
- Update terraform test to redact timings out.
- Clean up .workspace.current_user from outputs of the tests.
## Tests
Existing tests.
## Changes
- Include compact relPath in the error message title. Include full paths
in separate lines below.
- Previously sometimes full paths were printed, sometime only rel path.
## Tests
Manually trigger the errors.
It is proven to be not necessary.
```
~/work/cli/acceptance % hyperfine -w 2 'go test' # with change:
Benchmark 1: go test
Time (mean ± σ): 4.983 s ± 0.209 s [User: 6.073 s, System: 9.869 s]
Range (min … max): 4.792 s … 5.483 s 10 runs
~/work/cli/acceptance % git stash # without change:
~/work/cli/acceptance % hyperfine -w 2 'go test'
Benchmark 1: go test
Time (mean ± σ): 5.018 s ± 0.100 s [User: 6.142 s, System: 10.234 s]
Range (min … max): 4.899 s … 5.182 s 10 runs
```
## Changes
- Add a script install_terraform.py that downloads terraform and
provider and generates a config to use, inspired by
https://gist.github.com/pietern/1cb6b6f3e0a452328e13cdc75031105e
- Make acceptance tests run this script once before running the tests
and set the required env vars to make cli use this terraform
installation.
- Use OS-specific directory for things that are build by acceptance test
runner (CLI and terraform).
This enables acceptance tests against cloud #2242 and local test for
bundle deploy #2254.
## Tests
- Add an acceptance test for standalone terraform. This is useful to
debug terraform with TF_LOG=DEBUG to see that it uses local provider.
- Other acceptance tests are updated with regard to terraform exec path.
- The overall time for tests locally is unchanged (if terraform is
already fetched).
## Changes
With this PR, any acceptance tests that define custom server stubs in
`test.toml` will automatically record all HTTP requests made and assert
on them.
Builds on top of https://github.com/databricks/cli/pull/2226
## Tests
Modifying existing acceptance test.
## Changes
This PR allows us to define custom server stubs in a `test.toml` file.
Note: A followup PR will add functionality to do assertions on the API
request itself.
## Tests
New acceptance test.
## Changes
This PR registers the `server.Close()` function to be run during test
cleanup in the server initialization function. This ensures that all
test servers are closed as soon as the test they are scoped to finish.
Motivated by https://github.com/databricks/cli/pull/2255/files where a
regression was introduced where we did not close the test server.
## Tests
N/A
## Changes
Added support for double underscore variable references.
Previously we made this restriction stronger with no particular reason,
TF provider supports multiple underscores and thus DABs should do as
well.
Fixes#1753
## Tests
Added acceptance and integration tests
## Changes
Noticed this when working on
https://github.com/databricks/cli/pull/2221. `<` is a special HTML
character that is encoded during text replacement when using
`AssertEqualTexts`.
## Tests
N/A
## Changes
- If CLOUD_ENV is set to do not override with dummy value. This allows
running acceptance tests as integration tests.
- Needed for https://github.com/databricks/cli/pull/2242
## Tests
Manually run the test suite against dogfood. `CLOUD_ENV=aws go test
./acceptance`
## Changes
- Do not start replacement / comparison if file is too large or not
valid utf-8.
- This helps to prevent replacements if there is accidentally a large
binary (e.g. terraform).
## Tests
Found this problem when working on
https://github.com/databricks/cli/pull/2242 -- the tests tried to
applied replacements on terraform binary and crashed. With this change,
an error is reported instead.
## Changes
- Replace development version with $DEV_VERSION
- Update experimental-jobs-as-code to make use of it.
## Tests
- Existing tests.
- Using this in https://github.com/databricks/cli/pull/2213
There is a speed up in 0.5s but it is still 4.4s, so something else is
slow there.
Benchmarking bundle/templates/experimental-jobs-as-code:
```
# Without UV_CACHE_DIR
~/work/cli/acceptance/bundle/templates/experimental-jobs-as-code % hyperfine --warmup 2 'testme -count=1'
Benchmark 1: testme -count=1
Time (mean ± σ): 4.950 s ± 0.079 s [User: 2.730 s, System: 8.524 s]
Range (min … max): 4.838 s … 5.076 s 10 runs
# With UV_CACHE_DIR
~/work/cli/acceptance/bundle/templates/experimental-jobs-as-code % hyperfine --warmup 2 'testme -count=1'
Benchmark 1: testme -count=1
Time (mean ± σ): 4.410 s ± 0.049 s [User: 2.669 s, System: 8.710 s]
Range (min … max): 4.324 s … 4.467 s 10 runs
```
## Changes
- Ability to extend a list of replacements via test.toml
- Modify selftest to both demo this feature and to get rid of sed on
Windows.
## Tests
Acceptance tests. I'm also using it
https://github.com/databricks/cli/pull/2213 for things like pid.
## Changes
- File comparison files in acceptance test, print the contents of all
applied replacements. Do it once per test.
- Remove duplicate entries in replacement list.
## Tests
Manually, change out files of existing test, you'll get this printed
once, after first assertion:
```
acceptance_test.go:307: Available replacements:
REPL /Users/denis\.bilenko/work/cli/acceptance/build/databricks => $$CLI
REPL /private/var/folders/5y/9kkdnjw91p11vsqwk0cvmk200000gp/T/TestAccept598522733/001 => $$TMPHOME
...
```
## Changes
- New test covering failures in reading .git. One case results in error,
some result in warning (not shown).
- New helper withdir runs commands in a subdirectory.
## Tests
New acceptance test.
## Changes
- Acceptance tests load test.toml to configure test behaviour.
- If file is not found in the test directory, parents are searched,
until the test root.
- Currently there is one option: runtime.GOOS to switch off tests per
OS.
## Tests
Using it in https://github.com/databricks/cli/pull/2223 to disable test
on Windows that cannot be run there.
## Changes
When adding path, a few things should take care of:
- symlink expansion
- forward/backward slashes, so that tests could do sed 's/\\\\/\//g' to
make it pass on Windows (see
acceptance/bundle/syncroot/dotdot-git/script)
SetPath() function takes care of both.
This PR uses SetPath() on all paths consistently.
## Tests
Existing tests.
shreyas-goenka
Denis Bilenko
Andrew Nester
Gleb Kanterov
Simon Poltier
Ilya Kuznetsov