databricks-cli/acceptance
shreyas-goenka bf2aded8e9
Recover from panic gracefully (#2353)
## Changes
This PR adds a recovery function for panics. This indicates to all users
running into a panic that it's a bug and they should report it to
Databricks.

## Tests
Manually and acceptance test.

Before:
```
.venv➜  cli git:(panic-r) ✗ ./cli selftest panic                                                
panic: the databricks selftest panic command always panics

goroutine 1 [running]:
github.com/databricks/cli/cmd/selftest.New.newPanic.func1(0x1400016f208?, {0x1016ca925?, 0x4?, 0x1016ca929?})
        /Users/shreyas.goenka/cli2/cli/cmd/selftest/panic.go:9 +0x2c
github.com/spf13/cobra.(*Command).execute(0x1400016f208, {0x10279bc40, 0x0, 0x0})
        /Users/shreyas.goenka/cli2/cli/vendor/github.com/spf13/cobra/command.go:989 +0x81c
github.com/spf13/cobra.(*Command).ExecuteC(0x14000428908)
        /Users/shreyas.goenka/cli2/cli/vendor/github.com/spf13/cobra/command.go:1117 +0x344
github.com/spf13/cobra.(*Command).ExecuteContextC(...)
        /Users/shreyas.goenka/cli2/cli/vendor/github.com/spf13/cobra/command.go:1050
github.com/databricks/cli/cmd/root.Execute({0x101d60440?, 0x10279bc40?}, 0x10266dd78?)
        /Users/shreyas.goenka/cli2/cli/cmd/root/root.go:101 +0x58
main.main()
        /Users/shreyas.goenka/cli2/cli/main.go:13 +0x44
```

After:
```
.venv➜  cli git:(panic-r) ./cli selftest panic
The Databricks CLI unexpectedly had a fatal error.
Please report this issue to Databricks in the form of a GitHub issue at:
https://github.com/databricks/cli

CLI Version: 0.0.0-dev+aae7ced52d36

Panic Payload: the databricks selftest panic command always panics

Stack Trace:
goroutine 1 [running]:
runtime/debug.Stack()
        /Users/shreyas.goenka/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.23.4.darwin-arm64/src/runtime/debug/stack.go:26 +0x64
github.com/databricks/cli/cmd/root.Execute.func1()
        /Users/shreyas.goenka/cli2/cli/cmd/root/root.go:110 +0xa4
panic({0x10368b5e0?, 0x1039d6d70?})
        /Users/shreyas.goenka/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.23.4.darwin-arm64/src/runtime/panic.go:785 +0x124
github.com/databricks/cli/cmd/selftest.New.newPanic.func1(0x14000145208?, {0x103356be5?, 0x4?, 0x103356be9?})
        /Users/shreyas.goenka/cli2/cli/cmd/selftest/panic.go:9 +0x2c
github.com/spf13/cobra.(*Command).execute(0x14000145208, {0x104427c40, 0x0, 0x0})
        /Users/shreyas.goenka/cli2/cli/vendor/github.com/spf13/cobra/command.go:989 +0x81c
github.com/spf13/cobra.(*Command).ExecuteC(0x14000400c08)
        /Users/shreyas.goenka/cli2/cli/vendor/github.com/spf13/cobra/command.go:1117 +0x344
github.com/spf13/cobra.(*Command).ExecuteContextC(...)
        /Users/shreyas.goenka/cli2/cli/vendor/github.com/spf13/cobra/command.go:1050
github.com/databricks/cli/cmd/root.Execute({0x1039ec440?, 0x104427c40?}, 0x14000400c08)
        /Users/shreyas.goenka/cli2/cli/cmd/root/root.go:128 +0x94
main.main()
        /Users/shreyas.goenka/cli2/cli/main.go:13 +0x44
```
2025-02-27 13:27:36 +00:00
..
auth acc: replace LocalOnly option with Local & Cloud (#2387) 2025-02-26 16:01:49 +00:00
bin acc: Support multiline patterns in diff.py (#2384) 2025-02-26 14:25:39 +01:00
bundle Remove bundle.{Seq,If,Defer,newPhase,logString}, switch to regular functions (#2390) 2025-02-27 11:41:58 +00:00
cmd/workspace/apps acc: replace LocalOnly option with Local & Cloud (#2387) 2025-02-26 16:01:49 +00:00
help Fix duplicate "apps" entry in help output (#2191) 2025-01-20 16:02:29 +00:00
panic Recover from panic gracefully (#2353) 2025-02-27 13:27:36 +00:00
selftest acc: replace LocalOnly option with Local & Cloud (#2387) 2025-02-26 16:01:49 +00:00
terraform acc: replace LocalOnly option with Local & Cloud (#2387) 2025-02-26 16:01:49 +00:00
workspace/jobs acc: replace LocalOnly option with Local & Cloud (#2387) 2025-02-26 16:01:49 +00:00
.gitignore Use real terraform in acceptance tests (#2267) 2025-01-31 13:53:13 +00:00
README.md Add acceptance/selftest, showcasing basic features (#2229) 2025-01-27 09:17:22 +01:00
acceptance_test.go acc: replace LocalOnly option with Local & Cloud (#2387) 2025-02-26 16:01:49 +00:00
cmd_server_test.go acc: Simplify writing handlers; support headers in responses (#2338) 2025-02-12 13:00:57 +00:00
config_test.go acc: replace LocalOnly option with Local & Cloud (#2387) 2025-02-26 16:01:49 +00:00
install_terraform.py Use real terraform in acceptance tests (#2267) 2025-01-31 13:53:13 +00:00
script.cleanup Add acceptance tests (#2081) 2025-01-08 12:41:08 +00:00
script.prepare acc: Disable git hooks (#2249) 2025-01-28 14:00:41 +00:00
server_test.go Support serverless mode in default-python template (explicit prompt) (#2377) 2025-02-26 14:07:30 +01:00
test.toml acc: replace LocalOnly option with Local & Cloud (#2387) 2025-02-26 16:01:49 +00:00

README.md

Acceptance tests are blackbox tests that are run against compiled binary.

Currently these tests are run against "fake" HTTP server pretending to be Databricks API. However, they will be extended to run against real environment as regular integration tests.

To author a test,

  • Add a new directory under acceptance. Any level of nesting is supported.
  • Add databricks.yml there.
  • Add script with commands to run, e.g. $CLI bundle validate. The test case is recognized by presence of script.

The test runner will run script and capture output and compare it with output.txt file in the same directory.

In order to write output.txt for the first time or overwrite it with the current output pass -update flag to go test.

The scripts are run with bash -e so any errors will be propagated. They are captured in output.txt by appending Exit code: N line at the end.

For more complex tests one can also use:

  • errcode helper: if the command fails with non-zero code, it appends Exit code: N to the output but returns success to caller (bash), allowing continuation of script.
  • trace helper: prints the arguments before executing the command.
  • custom output files: redirect output to custom file (it must start with out), e.g. $CLI bundle validate > out.txt 2> out.error.txt.

See selftest for a toy test.