Commit Graph

207 Commits

Author SHA1 Message Date
Shreyas Goenka a407a5462d
fix test toml 2025-02-27 12:42:40 +01:00
Shreyas Goenka a556d51164
Merge remote-tracking branch 'origin' into async-logger-clean 2025-02-27 11:47:34 +01:00
Denis Bilenko 13ac52391d
acc: Include full output for default-python/classic (#2391)
## Tests
Include full output of default-python/classic so it can be used as a
base for diffs in cloud tests #2383
2025-02-27 10:16:06 +00:00
Denis Bilenko 81606cfcbc
acc: replace LocalOnly option with Local & Cloud (#2387)
## Changes
Instead of LocalOnly with non-composable semantics there are two
composable options:
- Local - enable test locally
- Cloud - enable test on the cloud

By default Cloud is switched off except in bundle (but not in
bundle/variables and bundle/help).

## Tests
Using this in #2383 to have test that runs on cloud but not locally.
2025-02-26 16:01:49 +00:00
Denis Bilenko 6d27c7e69d
acc: Support multiline patterns in diff.py (#2384)
## Changes
Fix diff.py to apply replacements that have newlines in them.

## Tests
Existing tests.
2025-02-26 14:25:39 +01:00
Denis Bilenko 03f2ff5a39
Support serverless mode in default-python template (explicit prompt) (#2377)
## Changes

- Add 'serverless' prompt to default-python template (default is
currently set to "no").
- This is a simplified version of
https://github.com/databricks/cli/pull/2348 with 'auto' functionality
removed.

## Tests
- Split default-python into default-python/classic,
default-python/serverless, default-python/serverless-customcatalog.
- Manually check that "bundle init default-python" with serverless=yes
can be deployed and run on dogfood and test env.
2025-02-26 14:07:30 +01:00
Andrew Nester df001dcdfe
Do not load host from bundle for CLI commands when profile flag is used (#2335)
## Changes
Now when `profile` flag is used we won't pick up host from bundle
anymore and use the one provided by -p flag

Previous behaviour in the context of bundle
```
databricks current-user me -p profile_name
Error: cannot resolve bundle auth configuration: config host mismatch: profile uses host https://non-existing-subdomain.databricks.com, but CLI configured to use https://foo.com
```

New behaviour (make an api call)
```
databricks current-user me -p profile_name
{
  email: "foo@bar.com"
  ...
}
```

We still load bundle configuration when `-t` flag provide because we
want to load host information from the target.

Fixes #1358 

## Tests
Added acceptance test
2025-02-26 12:30:38 +00:00
Anton Nekipelov 428e730c9e
Set default data_security_mode to "SINGLE_USER" in bundle templates (#2372)
## Changes
1. Change the **default-python** bundle template to set
`data_security_mode` of a cluster to SINGLE_USER
2. Change the **experimental-jobs-as-code** bundle template to set
`data_security_mode` of a cluster to SINGLE_USER

## Why
Explicitly adding this field saves experienced users from confusion onto
what security mode is applied to the cluster

## Tests
Changed existing unit and integration tests to pass with this change
2025-02-26 13:19:38 +01:00
Andrew Nester fa79d04980
acc: Added tests for deploying PyPi and Maven libraries (#2359)
## Changes
Added PyPi and Maven libraries tests

Needed for this PR since we don't currently have any coverage for PyPi
or Maven libraries
https://github.com/databricks/cli/pull/2382
2025-02-26 13:05:38 +01:00
Andrew Nester cdea775bd2
Fixed spark version check for clusters defined in the same bundle (#2374)
## Changes
Previously using python wheel tasks in the tasks with compute referering
to interactive cluster defied in the same bundle would produce a warning
like below

```
GET /api/2.1/clusters/get?cluster_id=${resources.clusters.development_cluster.id}
< HTTP/2.0 400 Bad Request
< {
<   "error_code": "INVALID_PARAMETER_VALUE",
<   "message": "Cluster ${resources.clusters.development_cluster.id} does not exist"
< } pid=14465 mutator=seq mutator=initialize mutator=seq mutator=PythonWrapperWarning sdk=true
```

This PR fixes it by making sure that we check spark version for such
clusters based on its bundle configuration and don't make API calls

## Tests
Added acceptance test
2025-02-26 13:04:45 +01:00
Andrew Nester 28e094c207
Fixed accessing required path parameters in CLI generation when --json flag (#2373)
## Changes
CLI generation template was using RequiredPathField from incorrect
request entity (body field from request and not request itself). Thus
for some of the commands required path parameters were not required when
--json was specified.

## Tests
Regenerated commands work correctly
2025-02-25 20:20:06 +00:00
Ilya Kuznetsov 25a701be92
Add missing `.gitignore` to dbt-sql and default-sql templates (#2356)
## Changes

Added missing .gitignore files to templates

## Tests

There were some incorrect snapshots of gitignore files in acceptance
tests, probably generated by testing infra. Updated them to new files

---------

Co-authored-by: Lennart Kats (databricks) <lennart.kats@databricks.com>
2025-02-25 09:42:02 +00:00
Denis Bilenko ccdbb1aeba
acc: fix override of settings with null values (#2369)
## Changes
Previously, one could not set `LocalOnly=true` in parent directory and
then override it with `LocalOnly=false` in child directory. This is
because, `false` is considered empty value by mergo.

In order to distinguish between 'explicitly set to false' and 'not set',
I've changed all simple variables in config to be pointers. Now, one can
always override those, because non-nil pointers are not null (with
mergo.WithoutDereference option).

## Tests
Manually:

```
~/work/cli/acceptance/bundle/templates/default-python % cat test.toml   # add this new file 
LocalOnly = false

~/work/cli/acceptance/bundle/templates/default-python % CLOUD_ENV=aws go test ../../.. -run ^TestAccept$/^bundle$/^templates$/^default-python$ -v
(the test is no longer skipped)
```
2025-02-25 08:53:46 +00:00
Shreyas Goenka 33a7f7151e
- 2025-02-24 18:31:43 +01:00
Shreyas Goenka 63ad316e8a
fx 2025-02-24 17:05:18 +01:00
Shreyas Goenka 6acb6f166e
consolidate repl 2025-02-24 17:03:23 +01:00
Shreyas Goenka 00ebbb9c47
add integration test 2025-02-24 16:59:17 +01:00
Shreyas Goenka 59396166c3
rename env var 2025-02-24 16:09:09 +01:00
Shreyas Goenka a0e6f8f635
pull latest 2025-02-24 15:56:05 +01:00
Shreyas Goenka 8cfbb335c8
Merge remote-tracking branch 'origin' into async-logger-clean 2025-02-24 15:55:17 +01:00
Shreyas Goenka 0d37654735
add test for timeout 2025-02-24 15:26:31 +01:00
Shreyas Goenka f11f5895ff
toml files 2025-02-24 14:54:39 +01:00
Denis Bilenko 4881fd873b
acc: Write out.requests.txt immediately (#2368)
## Changes
- Instead of collecting requests in memory and writing them at the end
of the test, write them right away. Then test authors can do filtering
with jq in 'script' or collect individual files per different command.
- testserver is now simpler - it just calls a caller-provided function.
The logging logic is moved to acceptance_test.go.

See https://github.com/databricks/cli/pull/2359/files#r1967591173

## Tests
Existing tests.
2025-02-24 13:37:05 +00:00
Lennart Kats (databricks) f99716b0a5
Remove `run_as` from the built-in templates (#2044)
## Changes

This removes the `run-as` property from the default templates. It's a
useful property but it still only works for jobs and it makes the
default databricks.yml a bit longer. It seems like users can just learn
about it from the docs and/or vary their deployment identity.

Depends on https://github.com/databricks/cli/pull/1712.
2025-02-24 08:31:46 +00:00
Shreyas Goenka 69f5b037e6
remove log line 2025-02-21 18:08:00 +01:00
Shreyas Goenka 68646a583d
remove leaking logs 2025-02-21 18:05:42 +01:00
Shreyas Goenka ae8ebaa757
add acc test for oauth' 2025-02-21 16:58:45 +01:00
Shreyas Goenka 5056070d11
remove user agent from telemetry tests 2025-02-21 16:48:30 +01:00
Shreyas Goenka d7abb3bd97
remove daemon acceptance tests 2025-02-21 15:56:28 +01:00
Shreyas Goenka 18748f7879
rename 2025-02-18 17:26:16 +01:00
Shreyas Goenka 3cfeb640e1
add skip functionality 2025-02-18 16:47:47 +01:00
Shreyas Goenka 520f06905b
use daemon 2025-02-18 16:30:50 +01:00
Shreyas Goenka f48fbb950b
merge daemon 2025-02-18 16:06:24 +01:00
Shreyas Goenka 8a0b359f8f
Merge remote-tracking branch 'origin' into async-logger-clean 2025-02-18 16:04:50 +01:00
Shreyas Goenka c59b11f7c1
- 2025-02-18 14:52:25 +01:00
Shreyas Goenka 608f0ad94f
- 2025-02-18 14:51:03 +01:00
Shreyas Goenka 8eb454da14
- 2025-02-18 14:50:02 +01:00
Shreyas Goenka f6bd82d68f
add readme 2025-02-18 14:49:24 +01:00
Shreyas Goenka f6a02237cf
extend timeout to 1 minute 2025-02-18 14:13:01 +01:00
Shreyas Goenka 50308f25a1
- 2025-02-18 13:17:14 +01:00
Shreyas Goenka a4009ed6fa
use script to wait 2025-02-18 13:14:29 +01:00
Shreyas Goenka 2c7126991b
- 2025-02-17 19:39:18 +01:00
Shreyas Goenka ecbbee1bb3
more test 2025-02-17 19:38:33 +01:00
Shreyas Goenka 3c4443c13f
cleanup 2025-02-17 19:27:53 +01:00
Shreyas Goenka e970a129af
- 2025-02-17 19:26:06 +01:00
Shreyas Goenka d15ff75f3b
- 2025-02-17 19:24:51 +01:00
Shreyas Goenka de6ebdb44e
Merge remote-tracking branch 'origin' into daemon-library 2025-02-17 19:24:31 +01:00
Shreyas Goenka 6148681ed1
implement 2025-02-17 19:23:02 +01:00
Lennart Kats (databricks) bc30d44097
Provide instructions for testing in the default-python template (#2355)
## Changes
Adds instructions for testing to the default-python template.

## Tests
- Unit & acceptance tests.
2025-02-17 12:38:03 +00:00
Denis Bilenko c0a56a93fb
acc: add a helper to diff with replacements (#2352)
## Changes

diff.py is like "diff -r -U2" but it applies replacements first to the
argument.

This allows comparing different output files and directories but ignore
differences that are going to be replaced by placeholders.

This is useful for tests that record large amount of files, specifically
"bundle init" with standard templates. In those tests, changing one
parameter results in a small diff so recording the full directory is not
helpful, because it's hard to see what changed there. I'm using it in
implementation of serverless mode for templates that need it: #2348 The
serverless templates are slightly different from classic, capturing the
diff helps to see exactly where.

Related small changes:
- Add [TESTROOT] replacement for absolute path to acceptance directory
in git repo.
- Add $TESTDIR env var for absolute path to a given test in git repo.

## Tests
- New test acceptance/selftest/diff to test the helper.
- Via #2348 which makes use of this feature.
2025-02-14 11:02:12 +00:00