Commit Graph

9 Commits

Author SHA1 Message Date
Denis Bilenko 5592fa889e
acceptance: add -buildvcs=false when building on Windows (#2148)
I get an error on my local Windows otherwise: error obtaining VCS
status: exit status 128

On CI this does not make a difference.
2025-01-15 10:45:57 +01:00
Denis Bilenko 55494a0bda
Add test about using variable in bundle.git.branch (#2118)
This test checks load git details functionality + variable interpolation
there.

The variables are not working there because LoadGitDetails mutator is
running before variable interpolation.

Additionally, correctly replace tmp path that is used for DATABRICKS_TF_EXEC_PATH
2025-01-15 10:34:51 +01:00
Denis Bilenko 6a7eefa54b
Use the same test names on Win as on other OSes (#2149)
## Changes
Use name format "TestAccept/bundle/variables/host" (previously slashes
were reversed on Windows).

## Tests
Manually run "go test ./acceptance -v -run
TestAccept/bundle/variables/host" in Windows VM.
2025-01-15 09:43:40 +01:00
Pieter Noordhuis 82e35530b0
Add acceptance tests for builtin templates (#2135)
## Changes

To accommodate:
* Add the server URL to the set of output replacements
* Include a call to the permissions API to the dummy server
* Run the main script in a subshell to isolate working directory changes
2025-01-14 18:23:34 +00:00
Denis Bilenko a5e09ab28a
Coverage for acceptance tests (#2123)
## Changes

Add two new make commands:
- make acc-cover: runs acceptance tests and outputs
coverage-acceptance.txt
- make acc-showcover: show coverage-acceptance.txt locally in browser

Using the GOCOVERDIR functionality:
https://go.dev/blog/integration-test-coverage

This works, but there are a couple of issues encountered:
- GOCOVERDIR does not play well with regular "go test -cover". Once this
fixed, we can simplify the code and have 'make cover' output coverage
for everything at once. We can also probably get rid of CLI_GOCOVERDIR.
https://github.com/golang/go/issues/66225
- When running tests in parallel to the same directory there is rare
conflict on writing covmeta file. For this reason each tests writes
coverage to their own directory which is then merged together by 'make
acc-cover'.

<!-- Summary of your changes that are easy to understand --


## Tests
Manually running the new make commands.
2025-01-14 14:19:00 +00:00
Denis Bilenko 2ae2b7e8c8
Enable acceptance tests for manually running against the cloud (#2120)
## Changes
- If CLOUD_ENV variable is set, acceptance will no longer set up server
& override DATABRICKS_HOST/DATABRICKS_TOKEN/HOME env vars.
- I've updated replacements logic in testdiff to use tester /
tester@databricks.com convention.

## Tests
Manually running current acceptance tests against dogfood on my laptop I
get all test pass except for 2 failures.

```
    --- FAIL: TestAccept/bundle/variables/env_overrides (0.09s)
    --- FAIL: TestAccept/bundle/variables/resolve-builtin (1.30s)
```
2025-01-14 13:50:28 +00:00
Denis Bilenko 6d3b4159bd
Log warnings to stderr for "bundle validate -o json" (#2109)
## Changes
Previously diagnostics were not seen in JSON output mode. This change
prints them to stderr.

This also fixes acceptance tests to preprocess all output with
s/execPath/$CLI/ not just output.txt.

## Tests
Existing acceptance tests. In one case I've added non-json command to
check that they match in output.
2025-01-10 08:51:59 +00:00
Denis Bilenko a0455bcaef
Migrate bundle/tests/undefined_resources_test.go to acceptance test (#2106)
Add sort_blocks.py helper to deal with non-determinism.
2025-01-09 15:21:24 +00:00
Denis Bilenko 185bbd28e4
Add acceptance tests (#2081)
## Changes
- New kind of test is added - acceptance tests. See acceptance/README.md
for explanation.
- A few tests are converted to acceptance tests by moving databricks.yml
to acceptance/ and adding corresponding script files.

As these tests run against compiled binary and can capture full output
of the command, they can be useful to support major changes such as
refactoring internal logging / diagnostics or complex variable
interpolation.

These are currently run as part of 'make test' but the intention is to
run them as part of integration tests as well.

### Benefits

- Full binary is tested, exactly as users get it.
  - We're not testing custom set of mutators like many existing tests.
- Not mocking anything, real SDK is used (although the HTTP endpoint is
not a real Databricks env).
- Easy to maintain: output can be updated automatically.
- Can easily set up external env, such as env vars, CLI args,
.databrickscfg location etc.

### Gaps

The tests currently share the test server and there is global place to
define handlers. We should have a way for tests to override / add new
handlers.

## Tests
I manually checked that output of new acceptance tests matches previous
asserts.
2025-01-08 12:41:08 +00:00