mirror of https://github.com/databricks/cli.git
6b1a778fe1
## Changes
Replace timestamps with fixed string before output is sorted (and before
test runner replacements are applied).
Otherwise the test sometimes fails with error below. Note, timestamps
themselves do not show it, because they were replaced.
```
--- FAIL: TestAccept/bundle/debug (0.78s)
acceptance_test.go:404: Diff:
--- bundle/debug/out.stderr.parallel.txt
+++ /var/folders/5y/9kkdnjw91p11vsqwk0cvmk200000gp/T/TestAcceptbundledebug1859985035/001/out.stderr.parallel.txt
@@ -8,8 +8,8 @@
10:07:59 Debug: ApplyReadOnly pid=12345 mutator=validate mutator (read-only)=parallel mutator (read-only)=validate:files_to_sync
10:07:59 Debug: ApplyReadOnly pid=12345 mutator=validate mutator (read-only)=parallel mutator (read-only)=validate:folder_permissions
10:07:59 Debug: ApplyReadOnly pid=12345 mutator=validate mutator (read-only)=parallel mutator (read-only)=validate:validate_sync_patterns
-10:07:59 Debug: Path /Workspace/Users/[USERNAME]/.bundle/debug/default/files has type directory (ID: 0) pid=12345 mutator=validate mutator (read-only)=parallel mutator (read-only)=validate:files_to_sync
10:07:59 Debug: non-retriable error: pid=12345 mutator=validate mutator (read-only)=parallel mutator (read-only)=validate:files_to_sync sdk=true
+10:07:59 Debug: Path /Workspace/Users/[USERNAME]/.bundle/debug/default/files has type directory (ID: 0) pid=12345 mutator=validate mutator (read-only)=parallel mutator (read-only)=validate:files_to_sync
< {} pid=12345 mutator=validate mutator (read-only)=parallel mutator (read-only)=validate:files_to_sync sdk=true
< {} pid=12345 mutator=validate mutator (read-only)=parallel mutator (read-only)=validate:files_to_sync sdk=true
< } pid=12345 mutator=validate mutator (read-only)=parallel mutator (read-only)=validate:files_to_sync sdk=true
```
## Tests
Running `hyperfine --min-runs 10 'go test ../.. -run
^TestAccept$/^bundle$/^debug$ -count=1' --show-output` detects flakiness
on main but not with this PR.
|
||
|---|---|---|
| .. | ||
| auth/bundle_and_profile | ||
| bin | ||
| bundle | ||
| help | ||
| selftest | ||
| terraform | ||
| workspace/jobs | ||
| .gitignore | ||
| README.md | ||
| acceptance_test.go | ||
| cmd_server_test.go | ||
| config_test.go | ||
| install_terraform.py | ||
| script.cleanup | ||
| script.prepare | ||
| server_test.go | ||
| test.toml | ||
README.md
Acceptance tests are blackbox tests that are run against compiled binary.
Currently these tests are run against "fake" HTTP server pretending to be Databricks API. However, they will be extended to run against real environment as regular integration tests.
To author a test,
- Add a new directory under
acceptance. Any level of nesting is supported. - Add
databricks.ymlthere. - Add
scriptwith commands to run, e.g.$CLI bundle validate. The test case is recognized by presence ofscript.
The test runner will run script and capture output and compare it with output.txt file in the same directory.
In order to write output.txt for the first time or overwrite it with the current output pass -update flag to go test.
The scripts are run with bash -e so any errors will be propagated. They are captured in output.txt by appending Exit code: N line at the end.
For more complex tests one can also use:
errcodehelper: if the command fails with non-zero code, it appendsExit code: Nto the output but returns success to caller (bash), allowing continuation of script.tracehelper: prints the arguments before executing the command.- custom output files: redirect output to custom file (it must start with
out), e.g.$CLI bundle validate > out.txt 2> out.error.txt.
See selftest for a toy test.