$VARNAME is what we use for environment variables, it's good to
separate.
Some people use envsubst for homemade variable interpolation, it's also
good to have separation there.
## Changes
- Include acceptance directory in integration tests. Acceptance tests
will not start local server if CLOUD_ENV is set, so they become
integration tests.
- Add dependency for vendor to integration, so that CLI can be build
there.
- Implement LocalOnly option in test.toml to opt out of running
acceptance tests as integration tests. Use it in certain tests that are
difficult or not necessary to fix when run as integration tests.
- Update terraform test to redact timings out.
- Clean up .workspace.current_user from outputs of the tests.
## Tests
Existing tests.
## Changes
- Add a script install_terraform.py that downloads terraform and
provider and generates a config to use, inspired by
https://gist.github.com/pietern/1cb6b6f3e0a452328e13cdc75031105e
- Make acceptance tests run this script once before running the tests
and set the required env vars to make cli use this terraform
installation.
- Use OS-specific directory for things that are build by acceptance test
runner (CLI and terraform).
This enables acceptance tests against cloud #2242 and local test for
bundle deploy #2254.
## Tests
- Add an acceptance test for standalone terraform. This is useful to
debug terraform with TF_LOG=DEBUG to see that it uses local provider.
- Other acceptance tests are updated with regard to terraform exec path.
- The overall time for tests locally is unchanged (if terraform is
already fetched).
## Changes
Added support for double underscore variable references.
Previously we made this restriction stronger with no particular reason,
TF provider supports multiple underscores and thus DABs should do as
well.
Fixes#1753
## Tests
Added acceptance and integration tests
## Changes
New source of default values for variables - variable file
`.databricks/bundle/<target>/variable-overrides.json`
CLI tries to stat and read that file every time during variable
initialisation phase
<!-- Summary of your changes that are easy to understand -->
## Tests
Acceptance tests
## Changes
- Instead of doing 2 passes on variable resolution, do a loop until
there are no more updates (or we reach count 100).
- Stacked on top of #2163 which is a regression test for this:
acceptance/bundle/variables/complex-transitive-deep
## Tests
Existing tests, new regression tests.
These tests already passed before, added for completeness:
- acceptance/bundle/variables/cycle
- acceptance/bundle/variables/complex-cross-ref
Follow up to #2157. That PR repeated variable resolution. This test
still does not resolve fully but would resolve with 3 passes. This is
slightly different from complex-transitive-deeper - this test does not
show any errors, the issue is purely not enough passes.
## Changes
- Remove ResolveVariableReferencesInComplexVariables - it blocked
complex-within-complex for no good reason.
- Repeat regular resolution twice, it helps with a couple test cases we
have.
There may be a case for running it 3 times or more in a loop, but there
is no test case for that, so this PR is simple incremental improvement.
## Tests
Existing acceptance tests. Previously all unit tests for complex
variables were converted to acceptance tests, to capture this change and
ensure nothing breaks.
This test checks load git details functionality + variable interpolation
there.
The variables are not working there because LoadGitDetails mutator is
running before variable interpolation.
Additionally, correctly replace tmp path that is used for DATABRICKS_TF_EXEC_PATH
## Changes
- If CLOUD_ENV variable is set, acceptance will no longer set up server
& override DATABRICKS_HOST/DATABRICKS_TOKEN/HOME env vars.
- I've updated replacements logic in testdiff to use tester /
tester@databricks.com convention.
## Tests
Manually running current acceptance tests against dogfood on my laptop I
get all test pass except for 2 failures.
```
--- FAIL: TestAccept/bundle/variables/env_overrides (0.09s)
--- FAIL: TestAccept/bundle/variables/resolve-builtin (1.30s)
```
## Changes
- New kind of test is added - acceptance tests. See acceptance/README.md
for explanation.
- A few tests are converted to acceptance tests by moving databricks.yml
to acceptance/ and adding corresponding script files.
As these tests run against compiled binary and can capture full output
of the command, they can be useful to support major changes such as
refactoring internal logging / diagnostics or complex variable
interpolation.
These are currently run as part of 'make test' but the intention is to
run them as part of integration tests as well.
### Benefits
- Full binary is tested, exactly as users get it.
- We're not testing custom set of mutators like many existing tests.
- Not mocking anything, real SDK is used (although the HTTP endpoint is
not a real Databricks env).
- Easy to maintain: output can be updated automatically.
- Can easily set up external env, such as env vars, CLI args,
.databrickscfg location etc.
### Gaps
The tests currently share the test server and there is global place to
define handlers. We should have a way for tests to override / add new
handlers.
## Tests
I manually checked that output of new acceptance tests matches previous
asserts.
Denis Bilenko
Andrew Nester
Ilya Kuznetsov