## Changes
Replacement was split between the type `ReplacementContext` and the
`ReplaceOutput` function. The latter also ran a couple of regular
expressions. This change consolidates them such that it is up to the
caller to compose the set of replacements to use.
This change is required to accommodate UUID replacement in #2146.
This test checks load git details functionality + variable interpolation
there.
The variables are not working there because LoadGitDetails mutator is
running before variable interpolation.
Additionally, correctly replace tmp path that is used for DATABRICKS_TF_EXEC_PATH
## Changes
Use name format "TestAccept/bundle/variables/host" (previously slashes
were reversed on Windows).
## Tests
Manually run "go test ./acceptance -v -run
TestAccept/bundle/variables/host" in Windows VM.
## Changes
To accommodate:
* Add the server URL to the set of output replacements
* Include a call to the permissions API to the dummy server
* Run the main script in a subshell to isolate working directory changes
## Changes
Add two new make commands:
- make acc-cover: runs acceptance tests and outputs
coverage-acceptance.txt
- make acc-showcover: show coverage-acceptance.txt locally in browser
Using the GOCOVERDIR functionality:
https://go.dev/blog/integration-test-coverage
This works, but there are a couple of issues encountered:
- GOCOVERDIR does not play well with regular "go test -cover". Once this
fixed, we can simplify the code and have 'make cover' output coverage
for everything at once. We can also probably get rid of CLI_GOCOVERDIR.
https://github.com/golang/go/issues/66225
- When running tests in parallel to the same directory there is rare
conflict on writing covmeta file. For this reason each tests writes
coverage to their own directory which is then merged together by 'make
acc-cover'.
<!-- Summary of your changes that are easy to understand --
## Tests
Manually running the new make commands.
## Changes
- If CLOUD_ENV variable is set, acceptance will no longer set up server
& override DATABRICKS_HOST/DATABRICKS_TOKEN/HOME env vars.
- I've updated replacements logic in testdiff to use tester /
tester@databricks.com convention.
## Tests
Manually running current acceptance tests against dogfood on my laptop I
get all test pass except for 2 failures.
```
--- FAIL: TestAccept/bundle/variables/env_overrides (0.09s)
--- FAIL: TestAccept/bundle/variables/resolve-builtin (1.30s)
```
## Changes
Move mutator.Merge{JobClusters,JobParameters,JobTasks,PipelineClusters}
after variable resolution. This helps with the case when key contains a
variable.
@pietern mentioned here
https://github.com/databricks/cli/pull/2101#pullrequestreview-2539168762
it should be safe.
## Tests
Existing acceptance that was capturing the bug is updated with corrected
output.
## Changes
Previously diagnostics were not seen in JSON output mode. This change
prints them to stderr.
This also fixes acceptance tests to preprocess all output with
s/execPath/$CLI/ not just output.txt.
## Tests
Existing acceptance tests. In one case I've added non-json command to
check that they match in output.
Since merge happens first, before variable resolution, the two jobs are
seen as different.
I also updated override/job_cluster/script to include more of the
output.
It's easier to remember and type and also validated and part of help:
```
% go test ./acceptance -updat 2>&1 | grep updat
flag provided but not defined: -updat
-update
```
## Changes
- New kind of test is added - acceptance tests. See acceptance/README.md
for explanation.
- A few tests are converted to acceptance tests by moving databricks.yml
to acceptance/ and adding corresponding script files.
As these tests run against compiled binary and can capture full output
of the command, they can be useful to support major changes such as
refactoring internal logging / diagnostics or complex variable
interpolation.
These are currently run as part of 'make test' but the intention is to
run them as part of integration tests as well.
### Benefits
- Full binary is tested, exactly as users get it.
- We're not testing custom set of mutators like many existing tests.
- Not mocking anything, real SDK is used (although the HTTP endpoint is
not a real Databricks env).
- Easy to maintain: output can be updated automatically.
- Can easily set up external env, such as env vars, CLI args,
.databrickscfg location etc.
### Gaps
The tests currently share the test server and there is global place to
define handlers. We should have a way for tests to override / add new
handlers.
## Tests
I manually checked that output of new acceptance tests matches previous
asserts.
Denis Bilenko
Pieter Noordhuis