steve/databricks-cli
steve
/
databricks-cli
mirror of https://github.com/databricks/cli.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
databricks-cli/bundle/render/render_text_output_test.go

576 lines
14 KiB
Go
Raw Normal View History

Improve `bundle validate` output (#1532) ## Changes This combination of changes allows pretty-printing errors happening during the "load" and "init" phases, including their locations. Move to render code into a separate module dedicated to rendering `diag.Diagnostics` in a human-readable format. This will be used for the `bundle deploy` command. Preserve the "bundle" value if an error occurs in mutators. Rewrite the go templates to handle the case when the bundle isn't yet loaded if an error occurs during loading, that is possible now. Improve rendering for errors and warnings: - don't render empty locations - render "details" for errors if they exist Add `root.ErrAlreadyPrinted` indicating that the error was already printed, and the CLI entry point shouldn't print it again. ## Tests Add tests for output, that are especially handy to detect extra newlines
2024-07-01 09:01:10 +00:00
package render
import (
"bytes"
Add a textual bundle summary command
2024-08-24 18:48:36 +00:00
"context"
"io"
Improve `bundle validate` output (#1532) ## Changes This combination of changes allows pretty-printing errors happening during the "load" and "init" phases, including their locations. Move to render code into a separate module dedicated to rendering `diag.Diagnostics` in a human-readable format. This will be used for the `bundle deploy` command. Preserve the "bundle" value if an error occurs in mutators. Rewrite the go templates to handle the case when the bundle isn't yet loaded if an error occurs during loading, that is possible now. Improve rendering for errors and warnings: - don't render empty locations - render "details" for errors if they exist Add `root.ErrAlreadyPrinted` indicating that the error was already printed, and the CLI entry point shouldn't print it again. ## Tests Add tests for output, that are especially handy to detect extra newlines
2024-07-01 09:01:10 +00:00
"testing"
"github.com/databricks/cli/bundle"
"github.com/databricks/cli/bundle/config"
Add a textual bundle summary command
2024-08-24 18:48:36 +00:00
"github.com/databricks/cli/bundle/config/resources"
Improve `bundle validate` output (#1532) ## Changes This combination of changes allows pretty-printing errors happening during the "load" and "init" phases, including their locations. Move to render code into a separate module dedicated to rendering `diag.Diagnostics` in a human-readable format. This will be used for the `bundle deploy` command. Preserve the "bundle" value if an error occurs in mutators. Rewrite the go templates to handle the case when the bundle isn't yet loaded if an error occurs during loading, that is possible now. Improve rendering for errors and warnings: - don't render empty locations - render "details" for errors if they exist Add `root.ErrAlreadyPrinted` indicating that the error was already printed, and the CLI entry point shouldn't print it again. ## Tests Add tests for output, that are especially handy to detect extra newlines
2024-07-01 09:01:10 +00:00
"github.com/databricks/cli/libs/diag"
"github.com/databricks/cli/libs/dyn"
Add a textual bundle summary command
2024-08-24 18:48:36 +00:00
"github.com/databricks/databricks-sdk-go/service/catalog"
Improve `bundle validate` output (#1532) ## Changes This combination of changes allows pretty-printing errors happening during the "load" and "init" phases, including their locations. Move to render code into a separate module dedicated to rendering `diag.Diagnostics` in a human-readable format. This will be used for the `bundle deploy` command. Preserve the "bundle" value if an error occurs in mutators. Rewrite the go templates to handle the case when the bundle isn't yet loaded if an error occurs during loading, that is possible now. Improve rendering for errors and warnings: - don't render empty locations - render "details" for errors if they exist Add `root.ErrAlreadyPrinted` indicating that the error was already printed, and the CLI entry point shouldn't print it again. ## Tests Add tests for output, that are especially handy to detect extra newlines
2024-07-01 09:01:10 +00:00
"github.com/databricks/databricks-sdk-go/service/iam"
Add a textual bundle summary command
2024-08-24 18:48:36 +00:00
"github.com/databricks/databricks-sdk-go/service/jobs"
"github.com/databricks/databricks-sdk-go/service/pipelines"
Improve `bundle validate` output (#1532) ## Changes This combination of changes allows pretty-printing errors happening during the "load" and "init" phases, including their locations. Move to render code into a separate module dedicated to rendering `diag.Diagnostics` in a human-readable format. This will be used for the `bundle deploy` command. Preserve the "bundle" value if an error occurs in mutators. Rewrite the go templates to handle the case when the bundle isn't yet loaded if an error occurs during loading, that is possible now. Improve rendering for errors and warnings: - don't render empty locations - render "details" for errors if they exist Add `root.ErrAlreadyPrinted` indicating that the error was already printed, and the CLI entry point shouldn't print it again. ## Tests Add tests for output, that are especially handy to detect extra newlines
2024-07-01 09:01:10 +00:00
"github.com/stretchr/testify/require"
)
type renderTestOutputTestCase struct {
name string
bundle *bundle.Bundle
diags diag.Diagnostics
Print diagnostics in 'bundle deploy' (#1579) ## Changes Print diagnostics in 'bundle deploy' similar to 'bundle validate'. This way if a bundle has any errors or warnings, they are going to be easy to notice. NB: due to how we render errors, there is one extra trailing new line in output, preserved in examples below ## Example: No errors or warnings ``` % databricks bundle deploy Building default... Deploying resources... Updating deployment state... Deployment complete! ``` ## Example: Error on load ``` % databricks bundle deploy Error: Databricks CLI version constraint not satisfied. Required: >= 1337.0.0, current: 0.0.0-dev ``` ## Example: Warning on load ``` % databricks bundle deploy Building default... Deploying resources... Updating deployment state... Deployment complete! Warning: unknown field: foo in databricks.yml:6:1 ``` ## Example: Error + warning on load ``` % databricks bundle deploy Warning: unknown field: foo in databricks.yml:6:1 Error: something went wrong ``` ## Example: Warning on load + error in init ``` % databricks bundle deploy Warning: unknown field: foo in databricks.yml:6:1 Error: Failed to xxx in yyy.yml Detailed explanation in multiple lines ``` ## Tests Tested manually
2024-07-10 11:14:57 +00:00
opts RenderOptions
Improve `bundle validate` output (#1532) ## Changes This combination of changes allows pretty-printing errors happening during the "load" and "init" phases, including their locations. Move to render code into a separate module dedicated to rendering `diag.Diagnostics` in a human-readable format. This will be used for the `bundle deploy` command. Preserve the "bundle" value if an error occurs in mutators. Rewrite the go templates to handle the case when the bundle isn't yet loaded if an error occurs during loading, that is possible now. Improve rendering for errors and warnings: - don't render empty locations - render "details" for errors if they exist Add `root.ErrAlreadyPrinted` indicating that the error was already printed, and the CLI entry point shouldn't print it again. ## Tests Add tests for output, that are especially handy to detect extra newlines
2024-07-01 09:01:10 +00:00
expected string
}
func TestRenderTextOutput(t *testing.T) {
loadingBundle := &bundle.Bundle{
Config: config.Root{
Bundle: config.Bundle{
Name: "test-bundle",
Target: "test-target",
},
},
}
testCases := []renderTestOutputTestCase{
{
name: "nil bundle and 1 error",
diags: diag.Diagnostics{
{
Severity: diag.Error,
Summary: "failed to load xxx",
},
},
Print diagnostics in 'bundle deploy' (#1579) ## Changes Print diagnostics in 'bundle deploy' similar to 'bundle validate'. This way if a bundle has any errors or warnings, they are going to be easy to notice. NB: due to how we render errors, there is one extra trailing new line in output, preserved in examples below ## Example: No errors or warnings ``` % databricks bundle deploy Building default... Deploying resources... Updating deployment state... Deployment complete! ``` ## Example: Error on load ``` % databricks bundle deploy Error: Databricks CLI version constraint not satisfied. Required: >= 1337.0.0, current: 0.0.0-dev ``` ## Example: Warning on load ``` % databricks bundle deploy Building default... Deploying resources... Updating deployment state... Deployment complete! Warning: unknown field: foo in databricks.yml:6:1 ``` ## Example: Error + warning on load ``` % databricks bundle deploy Warning: unknown field: foo in databricks.yml:6:1 Error: something went wrong ``` ## Example: Warning on load + error in init ``` % databricks bundle deploy Warning: unknown field: foo in databricks.yml:6:1 Error: Failed to xxx in yyy.yml Detailed explanation in multiple lines ``` ## Tests Tested manually
2024-07-10 11:14:57 +00:00
opts: RenderOptions{RenderSummaryTable: true},
Improve `bundle validate` output (#1532) ## Changes This combination of changes allows pretty-printing errors happening during the "load" and "init" phases, including their locations. Move to render code into a separate module dedicated to rendering `diag.Diagnostics` in a human-readable format. This will be used for the `bundle deploy` command. Preserve the "bundle" value if an error occurs in mutators. Rewrite the go templates to handle the case when the bundle isn't yet loaded if an error occurs during loading, that is possible now. Improve rendering for errors and warnings: - don't render empty locations - render "details" for errors if they exist Add `root.ErrAlreadyPrinted` indicating that the error was already printed, and the CLI entry point shouldn't print it again. ## Tests Add tests for output, that are especially handy to detect extra newlines
2024-07-01 09:01:10 +00:00
expected: "Error: failed to load xxx\n" +
"\n" +
"Found 1 error\n",
},
Add validation for files with a `.(resource-name).yml` extension (#1780) ## Changes We want to encourage a pattern of specifying only a single resource in a YAML file when the `.(resource-type).yml` extension is used (for example, `.job.yml`). This convention could allow us to bijectively map a resource YAML file to its corresponding resource in the Databricks workspace. This PR: 1. Emits a recommendation diagnostic when we detect this convention is being violated. We can promote this to a warning when we want to encourage this pattern more strongly. 2. Visualises the recommendation diagnostics in the `bundle validate` command. **NOTE:** While this PR also shows the recommendation for `.yaml` files, we do not encourage users to use this extension. We only support it here since it's part of the YAML standard and some existing users might already be using `.yaml`. ## Tests Unit tests and manually. Here's what an example output looks like: ``` Recommendation: define a single job in a file with the .job.yml extension. at resources.jobs.bar resources.jobs.foo in foo.job.yml:13:7 foo.job.yml:5:7 The following resources are defined or configured in this file: - bar (job) - foo (job) ``` --------- Co-authored-by: Lennart Kats (databricks) <lennart.kats@databricks.com>
2024-10-07 09:16:20 +00:00
{
name: "nil bundle and 1 recommendation",
diags: diag.Diagnostics{
{
Severity: diag.Recommendation,
Summary: "recommendation",
},
},
opts: RenderOptions{RenderSummaryTable: true},
expected: "Recommendation: recommendation\n" +
"\n" +
"Found 1 recommendation\n",
},
Improve `bundle validate` output (#1532) ## Changes This combination of changes allows pretty-printing errors happening during the "load" and "init" phases, including their locations. Move to render code into a separate module dedicated to rendering `diag.Diagnostics` in a human-readable format. This will be used for the `bundle deploy` command. Preserve the "bundle" value if an error occurs in mutators. Rewrite the go templates to handle the case when the bundle isn't yet loaded if an error occurs during loading, that is possible now. Improve rendering for errors and warnings: - don't render empty locations - render "details" for errors if they exist Add `root.ErrAlreadyPrinted` indicating that the error was already printed, and the CLI entry point shouldn't print it again. ## Tests Add tests for output, that are especially handy to detect extra newlines
2024-07-01 09:01:10 +00:00
{
name: "bundle during 'load' and 1 error",
bundle: loadingBundle,
diags: diag.Errorf("failed to load bundle"),
Print diagnostics in 'bundle deploy' (#1579) ## Changes Print diagnostics in 'bundle deploy' similar to 'bundle validate'. This way if a bundle has any errors or warnings, they are going to be easy to notice. NB: due to how we render errors, there is one extra trailing new line in output, preserved in examples below ## Example: No errors or warnings ``` % databricks bundle deploy Building default... Deploying resources... Updating deployment state... Deployment complete! ``` ## Example: Error on load ``` % databricks bundle deploy Error: Databricks CLI version constraint not satisfied. Required: >= 1337.0.0, current: 0.0.0-dev ``` ## Example: Warning on load ``` % databricks bundle deploy Building default... Deploying resources... Updating deployment state... Deployment complete! Warning: unknown field: foo in databricks.yml:6:1 ``` ## Example: Error + warning on load ``` % databricks bundle deploy Warning: unknown field: foo in databricks.yml:6:1 Error: something went wrong ``` ## Example: Warning on load + error in init ``` % databricks bundle deploy Warning: unknown field: foo in databricks.yml:6:1 Error: Failed to xxx in yyy.yml Detailed explanation in multiple lines ``` ## Tests Tested manually
2024-07-10 11:14:57 +00:00
opts: RenderOptions{RenderSummaryTable: true},
Improve `bundle validate` output (#1532) ## Changes This combination of changes allows pretty-printing errors happening during the "load" and "init" phases, including their locations. Move to render code into a separate module dedicated to rendering `diag.Diagnostics` in a human-readable format. This will be used for the `bundle deploy` command. Preserve the "bundle" value if an error occurs in mutators. Rewrite the go templates to handle the case when the bundle isn't yet loaded if an error occurs during loading, that is possible now. Improve rendering for errors and warnings: - don't render empty locations - render "details" for errors if they exist Add `root.ErrAlreadyPrinted` indicating that the error was already printed, and the CLI entry point shouldn't print it again. ## Tests Add tests for output, that are especially handy to detect extra newlines
2024-07-01 09:01:10 +00:00
expected: "Error: failed to load bundle\n" +
"\n" +
"Name: test-bundle\n" +
"Target: test-target\n" +
"\n" +
"Found 1 error\n",
},
{
name: "bundle during 'load' and 1 warning",
bundle: loadingBundle,
diags: diag.Warningf("failed to load bundle"),
Print diagnostics in 'bundle deploy' (#1579) ## Changes Print diagnostics in 'bundle deploy' similar to 'bundle validate'. This way if a bundle has any errors or warnings, they are going to be easy to notice. NB: due to how we render errors, there is one extra trailing new line in output, preserved in examples below ## Example: No errors or warnings ``` % databricks bundle deploy Building default... Deploying resources... Updating deployment state... Deployment complete! ``` ## Example: Error on load ``` % databricks bundle deploy Error: Databricks CLI version constraint not satisfied. Required: >= 1337.0.0, current: 0.0.0-dev ``` ## Example: Warning on load ``` % databricks bundle deploy Building default... Deploying resources... Updating deployment state... Deployment complete! Warning: unknown field: foo in databricks.yml:6:1 ``` ## Example: Error + warning on load ``` % databricks bundle deploy Warning: unknown field: foo in databricks.yml:6:1 Error: something went wrong ``` ## Example: Warning on load + error in init ``` % databricks bundle deploy Warning: unknown field: foo in databricks.yml:6:1 Error: Failed to xxx in yyy.yml Detailed explanation in multiple lines ``` ## Tests Tested manually
2024-07-10 11:14:57 +00:00
opts: RenderOptions{RenderSummaryTable: true},
Improve `bundle validate` output (#1532) ## Changes This combination of changes allows pretty-printing errors happening during the "load" and "init" phases, including their locations. Move to render code into a separate module dedicated to rendering `diag.Diagnostics` in a human-readable format. This will be used for the `bundle deploy` command. Preserve the "bundle" value if an error occurs in mutators. Rewrite the go templates to handle the case when the bundle isn't yet loaded if an error occurs during loading, that is possible now. Improve rendering for errors and warnings: - don't render empty locations - render "details" for errors if they exist Add `root.ErrAlreadyPrinted` indicating that the error was already printed, and the CLI entry point shouldn't print it again. ## Tests Add tests for output, that are especially handy to detect extra newlines
2024-07-01 09:01:10 +00:00
expected: "Warning: failed to load bundle\n" +
"\n" +
"Name: test-bundle\n" +
"Target: test-target\n" +
"\n" +
"Found 1 warning\n",
},
{
name: "bundle during 'load' and 2 warnings",
bundle: loadingBundle,
diags: diag.Warningf("warning (1)").Extend(diag.Warningf("warning (2)")),
Print diagnostics in 'bundle deploy' (#1579) ## Changes Print diagnostics in 'bundle deploy' similar to 'bundle validate'. This way if a bundle has any errors or warnings, they are going to be easy to notice. NB: due to how we render errors, there is one extra trailing new line in output, preserved in examples below ## Example: No errors or warnings ``` % databricks bundle deploy Building default... Deploying resources... Updating deployment state... Deployment complete! ``` ## Example: Error on load ``` % databricks bundle deploy Error: Databricks CLI version constraint not satisfied. Required: >= 1337.0.0, current: 0.0.0-dev ``` ## Example: Warning on load ``` % databricks bundle deploy Building default... Deploying resources... Updating deployment state... Deployment complete! Warning: unknown field: foo in databricks.yml:6:1 ``` ## Example: Error + warning on load ``` % databricks bundle deploy Warning: unknown field: foo in databricks.yml:6:1 Error: something went wrong ``` ## Example: Warning on load + error in init ``` % databricks bundle deploy Warning: unknown field: foo in databricks.yml:6:1 Error: Failed to xxx in yyy.yml Detailed explanation in multiple lines ``` ## Tests Tested manually
2024-07-10 11:14:57 +00:00
opts: RenderOptions{RenderSummaryTable: true},
Improve `bundle validate` output (#1532) ## Changes This combination of changes allows pretty-printing errors happening during the "load" and "init" phases, including their locations. Move to render code into a separate module dedicated to rendering `diag.Diagnostics` in a human-readable format. This will be used for the `bundle deploy` command. Preserve the "bundle" value if an error occurs in mutators. Rewrite the go templates to handle the case when the bundle isn't yet loaded if an error occurs during loading, that is possible now. Improve rendering for errors and warnings: - don't render empty locations - render "details" for errors if they exist Add `root.ErrAlreadyPrinted` indicating that the error was already printed, and the CLI entry point shouldn't print it again. ## Tests Add tests for output, that are especially handy to detect extra newlines
2024-07-01 09:01:10 +00:00
expected: "Warning: warning (1)\n" +
"\n" +
"Warning: warning (2)\n" +
"\n" +
"Name: test-bundle\n" +
"Target: test-target\n" +
"\n" +
"Found 2 warnings\n",
},
{
Add validation for files with a `.(resource-name).yml` extension (#1780) ## Changes We want to encourage a pattern of specifying only a single resource in a YAML file when the `.(resource-type).yml` extension is used (for example, `.job.yml`). This convention could allow us to bijectively map a resource YAML file to its corresponding resource in the Databricks workspace. This PR: 1. Emits a recommendation diagnostic when we detect this convention is being violated. We can promote this to a warning when we want to encourage this pattern more strongly. 2. Visualises the recommendation diagnostics in the `bundle validate` command. **NOTE:** While this PR also shows the recommendation for `.yaml` files, we do not encourage users to use this extension. We only support it here since it's part of the YAML standard and some existing users might already be using `.yaml`. ## Tests Unit tests and manually. Here's what an example output looks like: ``` Recommendation: define a single job in a file with the .job.yml extension. at resources.jobs.bar resources.jobs.foo in foo.job.yml:13:7 foo.job.yml:5:7 The following resources are defined or configured in this file: - bar (job) - foo (job) ``` --------- Co-authored-by: Lennart Kats (databricks) <lennart.kats@databricks.com>
2024-10-07 09:16:20 +00:00
name: "bundle during 'load' and 2 errors, 1 warning and 1 recommendation with details",
Improve `bundle validate` output (#1532) ## Changes This combination of changes allows pretty-printing errors happening during the "load" and "init" phases, including their locations. Move to render code into a separate module dedicated to rendering `diag.Diagnostics` in a human-readable format. This will be used for the `bundle deploy` command. Preserve the "bundle" value if an error occurs in mutators. Rewrite the go templates to handle the case when the bundle isn't yet loaded if an error occurs during loading, that is possible now. Improve rendering for errors and warnings: - don't render empty locations - render "details" for errors if they exist Add `root.ErrAlreadyPrinted` indicating that the error was already printed, and the CLI entry point shouldn't print it again. ## Tests Add tests for output, that are especially handy to detect extra newlines
2024-07-01 09:01:10 +00:00
bundle: loadingBundle,
diags: diag.Diagnostics{
diag.Diagnostic{
Support multiple locations for diagnostics (#1610) ## Changes This PR changes `diag.Diagnostics` to allow including multiple locations associated with the diagnostic message. The diagnostics that now return multiple locations with this PR are: 1. Warning for unknown keys in config. 2. Use of experimental.run_as 3. Accidental sync.exludes that exclude all files. ## Tests Existing unit tests pass. New unit test case to assert on error message when multiple locations are included. Example output: ``` ➜ bundle-playground-2 ~/cli2/cli/cli bundle validate Warning: You are using the legacy mode of run_as. The support for this mode is experimental and might be removed in a future release of the CLI. In order to run the DLT pipelines in your DAB as the run_as user this mode changes the owners of the pipelines to the run_as identity, which requires the user deploying the bundle to be a workspace admin, and also a Metastore admin if the pipeline target is in UC. at experimental.use_legacy_run_as in resources.yml:10:22 databricks.yml:13:22 Name: fix run_if Target: default Workspace: User: shreyas.goenka@databricks.com Path: /Users/shreyas.goenka@databricks.com/.bundle/fix run_if/default Found 1 warning ```
2024-07-23 17:20:11 +00:00
Severity: diag.Error,
Summary: "error (1)",
Detail: "detail (1)",
Locations: []dyn.Location{{File: "foo.py", Line: 1, Column: 1}},
Improve `bundle validate` output (#1532) ## Changes This combination of changes allows pretty-printing errors happening during the "load" and "init" phases, including their locations. Move to render code into a separate module dedicated to rendering `diag.Diagnostics` in a human-readable format. This will be used for the `bundle deploy` command. Preserve the "bundle" value if an error occurs in mutators. Rewrite the go templates to handle the case when the bundle isn't yet loaded if an error occurs during loading, that is possible now. Improve rendering for errors and warnings: - don't render empty locations - render "details" for errors if they exist Add `root.ErrAlreadyPrinted` indicating that the error was already printed, and the CLI entry point shouldn't print it again. ## Tests Add tests for output, that are especially handy to detect extra newlines
2024-07-01 09:01:10 +00:00
},
diag.Diagnostic{
Support multiple locations for diagnostics (#1610) ## Changes This PR changes `diag.Diagnostics` to allow including multiple locations associated with the diagnostic message. The diagnostics that now return multiple locations with this PR are: 1. Warning for unknown keys in config. 2. Use of experimental.run_as 3. Accidental sync.exludes that exclude all files. ## Tests Existing unit tests pass. New unit test case to assert on error message when multiple locations are included. Example output: ``` ➜ bundle-playground-2 ~/cli2/cli/cli bundle validate Warning: You are using the legacy mode of run_as. The support for this mode is experimental and might be removed in a future release of the CLI. In order to run the DLT pipelines in your DAB as the run_as user this mode changes the owners of the pipelines to the run_as identity, which requires the user deploying the bundle to be a workspace admin, and also a Metastore admin if the pipeline target is in UC. at experimental.use_legacy_run_as in resources.yml:10:22 databricks.yml:13:22 Name: fix run_if Target: default Workspace: User: shreyas.goenka@databricks.com Path: /Users/shreyas.goenka@databricks.com/.bundle/fix run_if/default Found 1 warning ```
2024-07-23 17:20:11 +00:00
Severity: diag.Error,
Summary: "error (2)",
Detail: "detail (2)",
Locations: []dyn.Location{{File: "foo.py", Line: 2, Column: 1}},
Improve `bundle validate` output (#1532) ## Changes This combination of changes allows pretty-printing errors happening during the "load" and "init" phases, including their locations. Move to render code into a separate module dedicated to rendering `diag.Diagnostics` in a human-readable format. This will be used for the `bundle deploy` command. Preserve the "bundle" value if an error occurs in mutators. Rewrite the go templates to handle the case when the bundle isn't yet loaded if an error occurs during loading, that is possible now. Improve rendering for errors and warnings: - don't render empty locations - render "details" for errors if they exist Add `root.ErrAlreadyPrinted` indicating that the error was already printed, and the CLI entry point shouldn't print it again. ## Tests Add tests for output, that are especially handy to detect extra newlines
2024-07-01 09:01:10 +00:00
},
diag.Diagnostic{
Support multiple locations for diagnostics (#1610) ## Changes This PR changes `diag.Diagnostics` to allow including multiple locations associated with the diagnostic message. The diagnostics that now return multiple locations with this PR are: 1. Warning for unknown keys in config. 2. Use of experimental.run_as 3. Accidental sync.exludes that exclude all files. ## Tests Existing unit tests pass. New unit test case to assert on error message when multiple locations are included. Example output: ``` ➜ bundle-playground-2 ~/cli2/cli/cli bundle validate Warning: You are using the legacy mode of run_as. The support for this mode is experimental and might be removed in a future release of the CLI. In order to run the DLT pipelines in your DAB as the run_as user this mode changes the owners of the pipelines to the run_as identity, which requires the user deploying the bundle to be a workspace admin, and also a Metastore admin if the pipeline target is in UC. at experimental.use_legacy_run_as in resources.yml:10:22 databricks.yml:13:22 Name: fix run_if Target: default Workspace: User: shreyas.goenka@databricks.com Path: /Users/shreyas.goenka@databricks.com/.bundle/fix run_if/default Found 1 warning ```
2024-07-23 17:20:11 +00:00
Severity: diag.Warning,
Summary: "warning (3)",
Detail: "detail (3)",
Locations: []dyn.Location{{File: "foo.py", Line: 3, Column: 1}},
Improve `bundle validate` output (#1532) ## Changes This combination of changes allows pretty-printing errors happening during the "load" and "init" phases, including their locations. Move to render code into a separate module dedicated to rendering `diag.Diagnostics` in a human-readable format. This will be used for the `bundle deploy` command. Preserve the "bundle" value if an error occurs in mutators. Rewrite the go templates to handle the case when the bundle isn't yet loaded if an error occurs during loading, that is possible now. Improve rendering for errors and warnings: - don't render empty locations - render "details" for errors if they exist Add `root.ErrAlreadyPrinted` indicating that the error was already printed, and the CLI entry point shouldn't print it again. ## Tests Add tests for output, that are especially handy to detect extra newlines
2024-07-01 09:01:10 +00:00
},
Add validation for files with a `.(resource-name).yml` extension (#1780) ## Changes We want to encourage a pattern of specifying only a single resource in a YAML file when the `.(resource-type).yml` extension is used (for example, `.job.yml`). This convention could allow us to bijectively map a resource YAML file to its corresponding resource in the Databricks workspace. This PR: 1. Emits a recommendation diagnostic when we detect this convention is being violated. We can promote this to a warning when we want to encourage this pattern more strongly. 2. Visualises the recommendation diagnostics in the `bundle validate` command. **NOTE:** While this PR also shows the recommendation for `.yaml` files, we do not encourage users to use this extension. We only support it here since it's part of the YAML standard and some existing users might already be using `.yaml`. ## Tests Unit tests and manually. Here's what an example output looks like: ``` Recommendation: define a single job in a file with the .job.yml extension. at resources.jobs.bar resources.jobs.foo in foo.job.yml:13:7 foo.job.yml:5:7 The following resources are defined or configured in this file: - bar (job) - foo (job) ``` --------- Co-authored-by: Lennart Kats (databricks) <lennart.kats@databricks.com>
2024-10-07 09:16:20 +00:00
diag.Diagnostic{
Severity: diag.Recommendation,
Summary: "recommendation (4)",
Detail: "detail (4)",
Locations: []dyn.Location{{File: "foo.py", Line: 4, Column: 1}},
},
Improve `bundle validate` output (#1532) ## Changes This combination of changes allows pretty-printing errors happening during the "load" and "init" phases, including their locations. Move to render code into a separate module dedicated to rendering `diag.Diagnostics` in a human-readable format. This will be used for the `bundle deploy` command. Preserve the "bundle" value if an error occurs in mutators. Rewrite the go templates to handle the case when the bundle isn't yet loaded if an error occurs during loading, that is possible now. Improve rendering for errors and warnings: - don't render empty locations - render "details" for errors if they exist Add `root.ErrAlreadyPrinted` indicating that the error was already printed, and the CLI entry point shouldn't print it again. ## Tests Add tests for output, that are especially handy to detect extra newlines
2024-07-01 09:01:10 +00:00
},
Print diagnostics in 'bundle deploy' (#1579) ## Changes Print diagnostics in 'bundle deploy' similar to 'bundle validate'. This way if a bundle has any errors or warnings, they are going to be easy to notice. NB: due to how we render errors, there is one extra trailing new line in output, preserved in examples below ## Example: No errors or warnings ``` % databricks bundle deploy Building default... Deploying resources... Updating deployment state... Deployment complete! ``` ## Example: Error on load ``` % databricks bundle deploy Error: Databricks CLI version constraint not satisfied. Required: >= 1337.0.0, current: 0.0.0-dev ``` ## Example: Warning on load ``` % databricks bundle deploy Building default... Deploying resources... Updating deployment state... Deployment complete! Warning: unknown field: foo in databricks.yml:6:1 ``` ## Example: Error + warning on load ``` % databricks bundle deploy Warning: unknown field: foo in databricks.yml:6:1 Error: something went wrong ``` ## Example: Warning on load + error in init ``` % databricks bundle deploy Warning: unknown field: foo in databricks.yml:6:1 Error: Failed to xxx in yyy.yml Detailed explanation in multiple lines ``` ## Tests Tested manually
2024-07-10 11:14:57 +00:00
opts: RenderOptions{RenderSummaryTable: true},
Improve `bundle validate` output (#1532) ## Changes This combination of changes allows pretty-printing errors happening during the "load" and "init" phases, including their locations. Move to render code into a separate module dedicated to rendering `diag.Diagnostics` in a human-readable format. This will be used for the `bundle deploy` command. Preserve the "bundle" value if an error occurs in mutators. Rewrite the go templates to handle the case when the bundle isn't yet loaded if an error occurs during loading, that is possible now. Improve rendering for errors and warnings: - don't render empty locations - render "details" for errors if they exist Add `root.ErrAlreadyPrinted` indicating that the error was already printed, and the CLI entry point shouldn't print it again. ## Tests Add tests for output, that are especially handy to detect extra newlines
2024-07-01 09:01:10 +00:00
expected: "Error: error (1)\n" +
" in foo.py:1:1\n" +
"\n" +
"detail (1)\n" +
"\n" +
"Error: error (2)\n" +
" in foo.py:2:1\n" +
"\n" +
"detail (2)\n" +
"\n" +
"Warning: warning (3)\n" +
" in foo.py:3:1\n" +
"\n" +
"detail (3)\n" +
"\n" +
Add validation for files with a `.(resource-name).yml` extension (#1780) ## Changes We want to encourage a pattern of specifying only a single resource in a YAML file when the `.(resource-type).yml` extension is used (for example, `.job.yml`). This convention could allow us to bijectively map a resource YAML file to its corresponding resource in the Databricks workspace. This PR: 1. Emits a recommendation diagnostic when we detect this convention is being violated. We can promote this to a warning when we want to encourage this pattern more strongly. 2. Visualises the recommendation diagnostics in the `bundle validate` command. **NOTE:** While this PR also shows the recommendation for `.yaml` files, we do not encourage users to use this extension. We only support it here since it's part of the YAML standard and some existing users might already be using `.yaml`. ## Tests Unit tests and manually. Here's what an example output looks like: ``` Recommendation: define a single job in a file with the .job.yml extension. at resources.jobs.bar resources.jobs.foo in foo.job.yml:13:7 foo.job.yml:5:7 The following resources are defined or configured in this file: - bar (job) - foo (job) ``` --------- Co-authored-by: Lennart Kats (databricks) <lennart.kats@databricks.com>
2024-10-07 09:16:20 +00:00
"Recommendation: recommendation (4)\n" +
" in foo.py:4:1\n" +
"\n" +
"detail (4)\n" +
"\n" +
"Name: test-bundle\n" +
"Target: test-target\n" +
"\n" +
"Found 2 errors, 1 warning, and 1 recommendation\n",
},
{
name: "bundle during 'load' and 1 error and 1 warning",
bundle: loadingBundle,
diags: diag.Diagnostics{
diag.Diagnostic{
Severity: diag.Error,
Summary: "error (1)",
Detail: "detail (1)",
Locations: []dyn.Location{{File: "foo.py", Line: 1, Column: 1}},
},
diag.Diagnostic{
Severity: diag.Warning,
Summary: "warning (2)",
Detail: "detail (2)",
Locations: []dyn.Location{{File: "foo.py", Line: 2, Column: 1}},
},
},
opts: RenderOptions{RenderSummaryTable: true},
expected: "Error: error (1)\n" +
" in foo.py:1:1\n" +
"\n" +
"detail (1)\n" +
"\n" +
"Warning: warning (2)\n" +
" in foo.py:2:1\n" +
"\n" +
"detail (2)\n" +
"\n" +
Improve `bundle validate` output (#1532) ## Changes This combination of changes allows pretty-printing errors happening during the "load" and "init" phases, including their locations. Move to render code into a separate module dedicated to rendering `diag.Diagnostics` in a human-readable format. This will be used for the `bundle deploy` command. Preserve the "bundle" value if an error occurs in mutators. Rewrite the go templates to handle the case when the bundle isn't yet loaded if an error occurs during loading, that is possible now. Improve rendering for errors and warnings: - don't render empty locations - render "details" for errors if they exist Add `root.ErrAlreadyPrinted` indicating that the error was already printed, and the CLI entry point shouldn't print it again. ## Tests Add tests for output, that are especially handy to detect extra newlines
2024-07-01 09:01:10 +00:00
"Name: test-bundle\n" +
"Target: test-target\n" +
"\n" +
Add validation for files with a `.(resource-name).yml` extension (#1780) ## Changes We want to encourage a pattern of specifying only a single resource in a YAML file when the `.(resource-type).yml` extension is used (for example, `.job.yml`). This convention could allow us to bijectively map a resource YAML file to its corresponding resource in the Databricks workspace. This PR: 1. Emits a recommendation diagnostic when we detect this convention is being violated. We can promote this to a warning when we want to encourage this pattern more strongly. 2. Visualises the recommendation diagnostics in the `bundle validate` command. **NOTE:** While this PR also shows the recommendation for `.yaml` files, we do not encourage users to use this extension. We only support it here since it's part of the YAML standard and some existing users might already be using `.yaml`. ## Tests Unit tests and manually. Here's what an example output looks like: ``` Recommendation: define a single job in a file with the .job.yml extension. at resources.jobs.bar resources.jobs.foo in foo.job.yml:13:7 foo.job.yml:5:7 The following resources are defined or configured in this file: - bar (job) - foo (job) ``` --------- Co-authored-by: Lennart Kats (databricks) <lennart.kats@databricks.com>
2024-10-07 09:16:20 +00:00
"Found 1 error and 1 warning\n",
},
{
name: "bundle during 'load' and 1 errors, 2 warning and 2 recommendations with details",
bundle: loadingBundle,
diags: diag.Diagnostics{
diag.Diagnostic{
Severity: diag.Error,
Summary: "error (1)",
Detail: "detail (1)",
Locations: []dyn.Location{{File: "foo.py", Line: 1, Column: 1}},
},
diag.Diagnostic{
Severity: diag.Warning,
Summary: "warning (2)",
Detail: "detail (2)",
Locations: []dyn.Location{{File: "foo.py", Line: 2, Column: 1}},
},
diag.Diagnostic{
Severity: diag.Warning,
Summary: "warning (3)",
Detail: "detail (3)",
Locations: []dyn.Location{{File: "foo.py", Line: 3, Column: 1}},
},
diag.Diagnostic{
Severity: diag.Recommendation,
Summary: "recommendation (4)",
Detail: "detail (4)",
Locations: []dyn.Location{{File: "foo.py", Line: 4, Column: 1}},
},
diag.Diagnostic{
Severity: diag.Recommendation,
Summary: "recommendation (5)",
Detail: "detail (5)",
Locations: []dyn.Location{{File: "foo.py", Line: 5, Column: 1}},
},
},
opts: RenderOptions{RenderSummaryTable: true},
expected: "Error: error (1)\n" +
" in foo.py:1:1\n" +
"\n" +
"detail (1)\n" +
"\n" +
"Warning: warning (2)\n" +
" in foo.py:2:1\n" +
"\n" +
"detail (2)\n" +
"\n" +
"Warning: warning (3)\n" +
" in foo.py:3:1\n" +
"\n" +
"detail (3)\n" +
"\n" +
"Recommendation: recommendation (4)\n" +
" in foo.py:4:1\n" +
"\n" +
"detail (4)\n" +
"\n" +
"Recommendation: recommendation (5)\n" +
" in foo.py:5:1\n" +
"\n" +
"detail (5)\n" +
"\n" +
"Name: test-bundle\n" +
"Target: test-target\n" +
"\n" +
"Found 1 error, 2 warnings, and 2 recommendations\n",
Improve `bundle validate` output (#1532) ## Changes This combination of changes allows pretty-printing errors happening during the "load" and "init" phases, including their locations. Move to render code into a separate module dedicated to rendering `diag.Diagnostics` in a human-readable format. This will be used for the `bundle deploy` command. Preserve the "bundle" value if an error occurs in mutators. Rewrite the go templates to handle the case when the bundle isn't yet loaded if an error occurs during loading, that is possible now. Improve rendering for errors and warnings: - don't render empty locations - render "details" for errors if they exist Add `root.ErrAlreadyPrinted` indicating that the error was already printed, and the CLI entry point shouldn't print it again. ## Tests Add tests for output, that are especially handy to detect extra newlines
2024-07-01 09:01:10 +00:00
},
{
name: "bundle during 'init'",
bundle: &bundle.Bundle{
Config: config.Root{
Bundle: config.Bundle{
Name: "test-bundle",
Target: "test-target",
},
Workspace: config.Workspace{
Host: "https://localhost/",
CurrentUser: &config.User{
User: &iam.User{
UserName: "test-user",
},
},
RootPath: "/Users/test-user@databricks.com/.bundle/examples/test-target",
},
},
},
diags: nil,
Print diagnostics in 'bundle deploy' (#1579) ## Changes Print diagnostics in 'bundle deploy' similar to 'bundle validate'. This way if a bundle has any errors or warnings, they are going to be easy to notice. NB: due to how we render errors, there is one extra trailing new line in output, preserved in examples below ## Example: No errors or warnings ``` % databricks bundle deploy Building default... Deploying resources... Updating deployment state... Deployment complete! ``` ## Example: Error on load ``` % databricks bundle deploy Error: Databricks CLI version constraint not satisfied. Required: >= 1337.0.0, current: 0.0.0-dev ``` ## Example: Warning on load ``` % databricks bundle deploy Building default... Deploying resources... Updating deployment state... Deployment complete! Warning: unknown field: foo in databricks.yml:6:1 ``` ## Example: Error + warning on load ``` % databricks bundle deploy Warning: unknown field: foo in databricks.yml:6:1 Error: something went wrong ``` ## Example: Warning on load + error in init ``` % databricks bundle deploy Warning: unknown field: foo in databricks.yml:6:1 Error: Failed to xxx in yyy.yml Detailed explanation in multiple lines ``` ## Tests Tested manually
2024-07-10 11:14:57 +00:00
opts: RenderOptions{RenderSummaryTable: true},
Improve `bundle validate` output (#1532) ## Changes This combination of changes allows pretty-printing errors happening during the "load" and "init" phases, including their locations. Move to render code into a separate module dedicated to rendering `diag.Diagnostics` in a human-readable format. This will be used for the `bundle deploy` command. Preserve the "bundle" value if an error occurs in mutators. Rewrite the go templates to handle the case when the bundle isn't yet loaded if an error occurs during loading, that is possible now. Improve rendering for errors and warnings: - don't render empty locations - render "details" for errors if they exist Add `root.ErrAlreadyPrinted` indicating that the error was already printed, and the CLI entry point shouldn't print it again. ## Tests Add tests for output, that are especially handy to detect extra newlines
2024-07-01 09:01:10 +00:00
expected: "Name: test-bundle\n" +
"Target: test-target\n" +
"Workspace:\n" +
" Host: https://localhost/\n" +
" User: test-user\n" +
" Path: /Users/test-user@databricks.com/.bundle/examples/test-target\n" +
"\n" +
"Validation OK!\n",
},
Print diagnostics in 'bundle deploy' (#1579) ## Changes Print diagnostics in 'bundle deploy' similar to 'bundle validate'. This way if a bundle has any errors or warnings, they are going to be easy to notice. NB: due to how we render errors, there is one extra trailing new line in output, preserved in examples below ## Example: No errors or warnings ``` % databricks bundle deploy Building default... Deploying resources... Updating deployment state... Deployment complete! ``` ## Example: Error on load ``` % databricks bundle deploy Error: Databricks CLI version constraint not satisfied. Required: >= 1337.0.0, current: 0.0.0-dev ``` ## Example: Warning on load ``` % databricks bundle deploy Building default... Deploying resources... Updating deployment state... Deployment complete! Warning: unknown field: foo in databricks.yml:6:1 ``` ## Example: Error + warning on load ``` % databricks bundle deploy Warning: unknown field: foo in databricks.yml:6:1 Error: something went wrong ``` ## Example: Warning on load + error in init ``` % databricks bundle deploy Warning: unknown field: foo in databricks.yml:6:1 Error: Failed to xxx in yyy.yml Detailed explanation in multiple lines ``` ## Tests Tested manually
2024-07-10 11:14:57 +00:00
{
Add validation for files with a `.(resource-name).yml` extension (#1780) ## Changes We want to encourage a pattern of specifying only a single resource in a YAML file when the `.(resource-type).yml` extension is used (for example, `.job.yml`). This convention could allow us to bijectively map a resource YAML file to its corresponding resource in the Databricks workspace. This PR: 1. Emits a recommendation diagnostic when we detect this convention is being violated. We can promote this to a warning when we want to encourage this pattern more strongly. 2. Visualises the recommendation diagnostics in the `bundle validate` command. **NOTE:** While this PR also shows the recommendation for `.yaml` files, we do not encourage users to use this extension. We only support it here since it's part of the YAML standard and some existing users might already be using `.yaml`. ## Tests Unit tests and manually. Here's what an example output looks like: ``` Recommendation: define a single job in a file with the .job.yml extension. at resources.jobs.bar resources.jobs.foo in foo.job.yml:13:7 foo.job.yml:5:7 The following resources are defined or configured in this file: - bar (job) - foo (job) ``` --------- Co-authored-by: Lennart Kats (databricks) <lennart.kats@databricks.com>
2024-10-07 09:16:20 +00:00
name: "nil bundle without summary with 1 error, 1 warning and 1 recommendation",
Print diagnostics in 'bundle deploy' (#1579) ## Changes Print diagnostics in 'bundle deploy' similar to 'bundle validate'. This way if a bundle has any errors or warnings, they are going to be easy to notice. NB: due to how we render errors, there is one extra trailing new line in output, preserved in examples below ## Example: No errors or warnings ``` % databricks bundle deploy Building default... Deploying resources... Updating deployment state... Deployment complete! ``` ## Example: Error on load ``` % databricks bundle deploy Error: Databricks CLI version constraint not satisfied. Required: >= 1337.0.0, current: 0.0.0-dev ``` ## Example: Warning on load ``` % databricks bundle deploy Building default... Deploying resources... Updating deployment state... Deployment complete! Warning: unknown field: foo in databricks.yml:6:1 ``` ## Example: Error + warning on load ``` % databricks bundle deploy Warning: unknown field: foo in databricks.yml:6:1 Error: something went wrong ``` ## Example: Warning on load + error in init ``` % databricks bundle deploy Warning: unknown field: foo in databricks.yml:6:1 Error: Failed to xxx in yyy.yml Detailed explanation in multiple lines ``` ## Tests Tested manually
2024-07-10 11:14:57 +00:00
bundle: nil,
diags: diag.Diagnostics{
diag.Diagnostic{
Support multiple locations for diagnostics (#1610) ## Changes This PR changes `diag.Diagnostics` to allow including multiple locations associated with the diagnostic message. The diagnostics that now return multiple locations with this PR are: 1. Warning for unknown keys in config. 2. Use of experimental.run_as 3. Accidental sync.exludes that exclude all files. ## Tests Existing unit tests pass. New unit test case to assert on error message when multiple locations are included. Example output: ``` ➜ bundle-playground-2 ~/cli2/cli/cli bundle validate Warning: You are using the legacy mode of run_as. The support for this mode is experimental and might be removed in a future release of the CLI. In order to run the DLT pipelines in your DAB as the run_as user this mode changes the owners of the pipelines to the run_as identity, which requires the user deploying the bundle to be a workspace admin, and also a Metastore admin if the pipeline target is in UC. at experimental.use_legacy_run_as in resources.yml:10:22 databricks.yml:13:22 Name: fix run_if Target: default Workspace: User: shreyas.goenka@databricks.com Path: /Users/shreyas.goenka@databricks.com/.bundle/fix run_if/default Found 1 warning ```
2024-07-23 17:20:11 +00:00
Severity: diag.Error,
Summary: "error (1)",
Detail: "detail (1)",
Locations: []dyn.Location{{File: "foo.py", Line: 1, Column: 1}},
Print diagnostics in 'bundle deploy' (#1579) ## Changes Print diagnostics in 'bundle deploy' similar to 'bundle validate'. This way if a bundle has any errors or warnings, they are going to be easy to notice. NB: due to how we render errors, there is one extra trailing new line in output, preserved in examples below ## Example: No errors or warnings ``` % databricks bundle deploy Building default... Deploying resources... Updating deployment state... Deployment complete! ``` ## Example: Error on load ``` % databricks bundle deploy Error: Databricks CLI version constraint not satisfied. Required: >= 1337.0.0, current: 0.0.0-dev ``` ## Example: Warning on load ``` % databricks bundle deploy Building default... Deploying resources... Updating deployment state... Deployment complete! Warning: unknown field: foo in databricks.yml:6:1 ``` ## Example: Error + warning on load ``` % databricks bundle deploy Warning: unknown field: foo in databricks.yml:6:1 Error: something went wrong ``` ## Example: Warning on load + error in init ``` % databricks bundle deploy Warning: unknown field: foo in databricks.yml:6:1 Error: Failed to xxx in yyy.yml Detailed explanation in multiple lines ``` ## Tests Tested manually
2024-07-10 11:14:57 +00:00
},
diag.Diagnostic{
Support multiple locations for diagnostics (#1610) ## Changes This PR changes `diag.Diagnostics` to allow including multiple locations associated with the diagnostic message. The diagnostics that now return multiple locations with this PR are: 1. Warning for unknown keys in config. 2. Use of experimental.run_as 3. Accidental sync.exludes that exclude all files. ## Tests Existing unit tests pass. New unit test case to assert on error message when multiple locations are included. Example output: ``` ➜ bundle-playground-2 ~/cli2/cli/cli bundle validate Warning: You are using the legacy mode of run_as. The support for this mode is experimental and might be removed in a future release of the CLI. In order to run the DLT pipelines in your DAB as the run_as user this mode changes the owners of the pipelines to the run_as identity, which requires the user deploying the bundle to be a workspace admin, and also a Metastore admin if the pipeline target is in UC. at experimental.use_legacy_run_as in resources.yml:10:22 databricks.yml:13:22 Name: fix run_if Target: default Workspace: User: shreyas.goenka@databricks.com Path: /Users/shreyas.goenka@databricks.com/.bundle/fix run_if/default Found 1 warning ```
2024-07-23 17:20:11 +00:00
Severity: diag.Warning,
Summary: "warning (2)",
Detail: "detail (2)",
Locations: []dyn.Location{{File: "foo.py", Line: 3, Column: 1}},
Print diagnostics in 'bundle deploy' (#1579) ## Changes Print diagnostics in 'bundle deploy' similar to 'bundle validate'. This way if a bundle has any errors or warnings, they are going to be easy to notice. NB: due to how we render errors, there is one extra trailing new line in output, preserved in examples below ## Example: No errors or warnings ``` % databricks bundle deploy Building default... Deploying resources... Updating deployment state... Deployment complete! ``` ## Example: Error on load ``` % databricks bundle deploy Error: Databricks CLI version constraint not satisfied. Required: >= 1337.0.0, current: 0.0.0-dev ``` ## Example: Warning on load ``` % databricks bundle deploy Building default... Deploying resources... Updating deployment state... Deployment complete! Warning: unknown field: foo in databricks.yml:6:1 ``` ## Example: Error + warning on load ``` % databricks bundle deploy Warning: unknown field: foo in databricks.yml:6:1 Error: something went wrong ``` ## Example: Warning on load + error in init ``` % databricks bundle deploy Warning: unknown field: foo in databricks.yml:6:1 Error: Failed to xxx in yyy.yml Detailed explanation in multiple lines ``` ## Tests Tested manually
2024-07-10 11:14:57 +00:00
},
Add validation for files with a `.(resource-name).yml` extension (#1780) ## Changes We want to encourage a pattern of specifying only a single resource in a YAML file when the `.(resource-type).yml` extension is used (for example, `.job.yml`). This convention could allow us to bijectively map a resource YAML file to its corresponding resource in the Databricks workspace. This PR: 1. Emits a recommendation diagnostic when we detect this convention is being violated. We can promote this to a warning when we want to encourage this pattern more strongly. 2. Visualises the recommendation diagnostics in the `bundle validate` command. **NOTE:** While this PR also shows the recommendation for `.yaml` files, we do not encourage users to use this extension. We only support it here since it's part of the YAML standard and some existing users might already be using `.yaml`. ## Tests Unit tests and manually. Here's what an example output looks like: ``` Recommendation: define a single job in a file with the .job.yml extension. at resources.jobs.bar resources.jobs.foo in foo.job.yml:13:7 foo.job.yml:5:7 The following resources are defined or configured in this file: - bar (job) - foo (job) ``` --------- Co-authored-by: Lennart Kats (databricks) <lennart.kats@databricks.com>
2024-10-07 09:16:20 +00:00
diag.Diagnostic{
Severity: diag.Recommendation,
Summary: "recommendation (3)",
Detail: "detail (3)",
Locations: []dyn.Location{{File: "foo.py", Line: 5, Column: 1}},
},
Print diagnostics in 'bundle deploy' (#1579) ## Changes Print diagnostics in 'bundle deploy' similar to 'bundle validate'. This way if a bundle has any errors or warnings, they are going to be easy to notice. NB: due to how we render errors, there is one extra trailing new line in output, preserved in examples below ## Example: No errors or warnings ``` % databricks bundle deploy Building default... Deploying resources... Updating deployment state... Deployment complete! ``` ## Example: Error on load ``` % databricks bundle deploy Error: Databricks CLI version constraint not satisfied. Required: >= 1337.0.0, current: 0.0.0-dev ``` ## Example: Warning on load ``` % databricks bundle deploy Building default... Deploying resources... Updating deployment state... Deployment complete! Warning: unknown field: foo in databricks.yml:6:1 ``` ## Example: Error + warning on load ``` % databricks bundle deploy Warning: unknown field: foo in databricks.yml:6:1 Error: something went wrong ``` ## Example: Warning on load + error in init ``` % databricks bundle deploy Warning: unknown field: foo in databricks.yml:6:1 Error: Failed to xxx in yyy.yml Detailed explanation in multiple lines ``` ## Tests Tested manually
2024-07-10 11:14:57 +00:00
},
opts: RenderOptions{RenderSummaryTable: false},
expected: "Error: error (1)\n" +
" in foo.py:1:1\n" +
"\n" +
"detail (1)\n" +
"\n" +
"Warning: warning (2)\n" +
" in foo.py:3:1\n" +
"\n" +
"detail (2)\n" +
Add validation for files with a `.(resource-name).yml` extension (#1780) ## Changes We want to encourage a pattern of specifying only a single resource in a YAML file when the `.(resource-type).yml` extension is used (for example, `.job.yml`). This convention could allow us to bijectively map a resource YAML file to its corresponding resource in the Databricks workspace. This PR: 1. Emits a recommendation diagnostic when we detect this convention is being violated. We can promote this to a warning when we want to encourage this pattern more strongly. 2. Visualises the recommendation diagnostics in the `bundle validate` command. **NOTE:** While this PR also shows the recommendation for `.yaml` files, we do not encourage users to use this extension. We only support it here since it's part of the YAML standard and some existing users might already be using `.yaml`. ## Tests Unit tests and manually. Here's what an example output looks like: ``` Recommendation: define a single job in a file with the .job.yml extension. at resources.jobs.bar resources.jobs.foo in foo.job.yml:13:7 foo.job.yml:5:7 The following resources are defined or configured in this file: - bar (job) - foo (job) ``` --------- Co-authored-by: Lennart Kats (databricks) <lennart.kats@databricks.com>
2024-10-07 09:16:20 +00:00
"\n" +
"Recommendation: recommendation (3)\n" +
" in foo.py:5:1\n" +
"\n" +
"detail (3)\n" +
Print diagnostics in 'bundle deploy' (#1579) ## Changes Print diagnostics in 'bundle deploy' similar to 'bundle validate'. This way if a bundle has any errors or warnings, they are going to be easy to notice. NB: due to how we render errors, there is one extra trailing new line in output, preserved in examples below ## Example: No errors or warnings ``` % databricks bundle deploy Building default... Deploying resources... Updating deployment state... Deployment complete! ``` ## Example: Error on load ``` % databricks bundle deploy Error: Databricks CLI version constraint not satisfied. Required: >= 1337.0.0, current: 0.0.0-dev ``` ## Example: Warning on load ``` % databricks bundle deploy Building default... Deploying resources... Updating deployment state... Deployment complete! Warning: unknown field: foo in databricks.yml:6:1 ``` ## Example: Error + warning on load ``` % databricks bundle deploy Warning: unknown field: foo in databricks.yml:6:1 Error: something went wrong ``` ## Example: Warning on load + error in init ``` % databricks bundle deploy Warning: unknown field: foo in databricks.yml:6:1 Error: Failed to xxx in yyy.yml Detailed explanation in multiple lines ``` ## Tests Tested manually
2024-07-10 11:14:57 +00:00
"\n",
},
Improve `bundle validate` output (#1532) ## Changes This combination of changes allows pretty-printing errors happening during the "load" and "init" phases, including their locations. Move to render code into a separate module dedicated to rendering `diag.Diagnostics` in a human-readable format. This will be used for the `bundle deploy` command. Preserve the "bundle" value if an error occurs in mutators. Rewrite the go templates to handle the case when the bundle isn't yet loaded if an error occurs during loading, that is possible now. Improve rendering for errors and warnings: - don't render empty locations - render "details" for errors if they exist Add `root.ErrAlreadyPrinted` indicating that the error was already printed, and the CLI entry point shouldn't print it again. ## Tests Add tests for output, that are especially handy to detect extra newlines
2024-07-01 09:01:10 +00:00
}
for _, tc := range testCases {
t.Run(tc.name, func(t *testing.T) {
writer := &bytes.Buffer{}
Add a textual bundle summary command
2024-08-24 18:48:36 +00:00
err := RenderDiagnostics(writer, tc.bundle, tc.diags, tc.opts)
Improve `bundle validate` output (#1532) ## Changes This combination of changes allows pretty-printing errors happening during the "load" and "init" phases, including their locations. Move to render code into a separate module dedicated to rendering `diag.Diagnostics` in a human-readable format. This will be used for the `bundle deploy` command. Preserve the "bundle" value if an error occurs in mutators. Rewrite the go templates to handle the case when the bundle isn't yet loaded if an error occurs during loading, that is possible now. Improve rendering for errors and warnings: - don't render empty locations - render "details" for errors if they exist Add `root.ErrAlreadyPrinted` indicating that the error was already printed, and the CLI entry point shouldn't print it again. ## Tests Add tests for output, that are especially handy to detect extra newlines
2024-07-01 09:01:10 +00:00
require.NoError(t, err)
Add a textual bundle summary command
2024-08-24 18:48:36 +00:00
require.Equal(t, tc.expected, writer.String())
Improve `bundle validate` output (#1532) ## Changes This combination of changes allows pretty-printing errors happening during the "load" and "init" phases, including their locations. Move to render code into a separate module dedicated to rendering `diag.Diagnostics` in a human-readable format. This will be used for the `bundle deploy` command. Preserve the "bundle" value if an error occurs in mutators. Rewrite the go templates to handle the case when the bundle isn't yet loaded if an error occurs during loading, that is possible now. Improve rendering for errors and warnings: - don't render empty locations - render "details" for errors if they exist Add `root.ErrAlreadyPrinted` indicating that the error was already printed, and the CLI entry point shouldn't print it again. ## Tests Add tests for output, that are especially handy to detect extra newlines
2024-07-01 09:01:10 +00:00
})
}
}
type renderDiagnosticsTestCase struct {
name string
diags diag.Diagnostics
expected string
}
func TestRenderDiagnostics(t *testing.T) {
bundle := &bundle.Bundle{}
testCases := []renderDiagnosticsTestCase{
{
name: "empty diagnostics",
diags: diag.Diagnostics{},
expected: "",
},
{
name: "error with short summary",
diags: diag.Diagnostics{
{
Severity: diag.Error,
Summary: "failed to load xxx",
},
},
expected: "Error: failed to load xxx\n\n",
},
{
name: "error with source location",
diags: diag.Diagnostics{
{
Severity: diag.Error,
Summary: "failed to load xxx",
Detail: "'name' is required",
Support multiple locations for diagnostics (#1610) ## Changes This PR changes `diag.Diagnostics` to allow including multiple locations associated with the diagnostic message. The diagnostics that now return multiple locations with this PR are: 1. Warning for unknown keys in config. 2. Use of experimental.run_as 3. Accidental sync.exludes that exclude all files. ## Tests Existing unit tests pass. New unit test case to assert on error message when multiple locations are included. Example output: ``` ➜ bundle-playground-2 ~/cli2/cli/cli bundle validate Warning: You are using the legacy mode of run_as. The support for this mode is experimental and might be removed in a future release of the CLI. In order to run the DLT pipelines in your DAB as the run_as user this mode changes the owners of the pipelines to the run_as identity, which requires the user deploying the bundle to be a workspace admin, and also a Metastore admin if the pipeline target is in UC. at experimental.use_legacy_run_as in resources.yml:10:22 databricks.yml:13:22 Name: fix run_if Target: default Workspace: User: shreyas.goenka@databricks.com Path: /Users/shreyas.goenka@databricks.com/.bundle/fix run_if/default Found 1 warning ```
2024-07-23 17:20:11 +00:00
Locations: []dyn.Location{{
Improve `bundle validate` output (#1532) ## Changes This combination of changes allows pretty-printing errors happening during the "load" and "init" phases, including their locations. Move to render code into a separate module dedicated to rendering `diag.Diagnostics` in a human-readable format. This will be used for the `bundle deploy` command. Preserve the "bundle" value if an error occurs in mutators. Rewrite the go templates to handle the case when the bundle isn't yet loaded if an error occurs during loading, that is possible now. Improve rendering for errors and warnings: - don't render empty locations - render "details" for errors if they exist Add `root.ErrAlreadyPrinted` indicating that the error was already printed, and the CLI entry point shouldn't print it again. ## Tests Add tests for output, that are especially handy to detect extra newlines
2024-07-01 09:01:10 +00:00
File: "foo.yaml",
Line: 1,
Support multiple locations for diagnostics (#1610) ## Changes This PR changes `diag.Diagnostics` to allow including multiple locations associated with the diagnostic message. The diagnostics that now return multiple locations with this PR are: 1. Warning for unknown keys in config. 2. Use of experimental.run_as 3. Accidental sync.exludes that exclude all files. ## Tests Existing unit tests pass. New unit test case to assert on error message when multiple locations are included. Example output: ``` ➜ bundle-playground-2 ~/cli2/cli/cli bundle validate Warning: You are using the legacy mode of run_as. The support for this mode is experimental and might be removed in a future release of the CLI. In order to run the DLT pipelines in your DAB as the run_as user this mode changes the owners of the pipelines to the run_as identity, which requires the user deploying the bundle to be a workspace admin, and also a Metastore admin if the pipeline target is in UC. at experimental.use_legacy_run_as in resources.yml:10:22 databricks.yml:13:22 Name: fix run_if Target: default Workspace: User: shreyas.goenka@databricks.com Path: /Users/shreyas.goenka@databricks.com/.bundle/fix run_if/default Found 1 warning ```
2024-07-23 17:20:11 +00:00
Column: 2}},
Improve `bundle validate` output (#1532) ## Changes This combination of changes allows pretty-printing errors happening during the "load" and "init" phases, including their locations. Move to render code into a separate module dedicated to rendering `diag.Diagnostics` in a human-readable format. This will be used for the `bundle deploy` command. Preserve the "bundle" value if an error occurs in mutators. Rewrite the go templates to handle the case when the bundle isn't yet loaded if an error occurs during loading, that is possible now. Improve rendering for errors and warnings: - don't render empty locations - render "details" for errors if they exist Add `root.ErrAlreadyPrinted` indicating that the error was already printed, and the CLI entry point shouldn't print it again. ## Tests Add tests for output, that are especially handy to detect extra newlines
2024-07-01 09:01:10 +00:00
},
},
expected: "Error: failed to load xxx\n" +
" in foo.yaml:1:2\n\n" +
"'name' is required\n\n",
},
Support multiple locations for diagnostics (#1610) ## Changes This PR changes `diag.Diagnostics` to allow including multiple locations associated with the diagnostic message. The diagnostics that now return multiple locations with this PR are: 1. Warning for unknown keys in config. 2. Use of experimental.run_as 3. Accidental sync.exludes that exclude all files. ## Tests Existing unit tests pass. New unit test case to assert on error message when multiple locations are included. Example output: ``` ➜ bundle-playground-2 ~/cli2/cli/cli bundle validate Warning: You are using the legacy mode of run_as. The support for this mode is experimental and might be removed in a future release of the CLI. In order to run the DLT pipelines in your DAB as the run_as user this mode changes the owners of the pipelines to the run_as identity, which requires the user deploying the bundle to be a workspace admin, and also a Metastore admin if the pipeline target is in UC. at experimental.use_legacy_run_as in resources.yml:10:22 databricks.yml:13:22 Name: fix run_if Target: default Workspace: User: shreyas.goenka@databricks.com Path: /Users/shreyas.goenka@databricks.com/.bundle/fix run_if/default Found 1 warning ```
2024-07-23 17:20:11 +00:00
{
name: "error with multiple source locations",
diags: diag.Diagnostics{
{
Severity: diag.Error,
Summary: "failed to load xxx",
Detail: "'name' is required",
Locations: []dyn.Location{
{
File: "foo.yaml",
Line: 1,
Column: 2,
},
{
File: "bar.yaml",
Line: 3,
Column: 4,
},
},
},
},
expected: "Error: failed to load xxx\n" +
" in foo.yaml:1:2\n" +
" bar.yaml:3:4\n\n" +
"'name' is required\n\n",
},
Improve `bundle validate` output (#1532) ## Changes This combination of changes allows pretty-printing errors happening during the "load" and "init" phases, including their locations. Move to render code into a separate module dedicated to rendering `diag.Diagnostics` in a human-readable format. This will be used for the `bundle deploy` command. Preserve the "bundle" value if an error occurs in mutators. Rewrite the go templates to handle the case when the bundle isn't yet loaded if an error occurs during loading, that is possible now. Improve rendering for errors and warnings: - don't render empty locations - render "details" for errors if they exist Add `root.ErrAlreadyPrinted` indicating that the error was already printed, and the CLI entry point shouldn't print it again. ## Tests Add tests for output, that are especially handy to detect extra newlines
2024-07-01 09:01:10 +00:00
{
name: "error with path",
diags: diag.Diagnostics{
{
Severity: diag.Error,
Detail: "'name' is required",
Summary: "failed to load xxx",
Support multiple paths for diagnostics (#1616) ## Changes Some diagnostics can have multiple paths associated with them. For instance, ensuring that unique resource keys are used across all resources. This PR extends `diag.Diagnostic` to accept multiple paths. This PR is symmetrical to https://github.com/databricks/cli/pull/1610/files ## Tests Unit tests
2024-07-25 15:16:27 +00:00
Paths: []dyn.Path{dyn.MustPathFromString("resources.jobs.xxx")},
Improve `bundle validate` output (#1532) ## Changes This combination of changes allows pretty-printing errors happening during the "load" and "init" phases, including their locations. Move to render code into a separate module dedicated to rendering `diag.Diagnostics` in a human-readable format. This will be used for the `bundle deploy` command. Preserve the "bundle" value if an error occurs in mutators. Rewrite the go templates to handle the case when the bundle isn't yet loaded if an error occurs during loading, that is possible now. Improve rendering for errors and warnings: - don't render empty locations - render "details" for errors if they exist Add `root.ErrAlreadyPrinted` indicating that the error was already printed, and the CLI entry point shouldn't print it again. ## Tests Add tests for output, that are especially handy to detect extra newlines
2024-07-01 09:01:10 +00:00
},
},
expected: "Error: failed to load xxx\n" +
" at resources.jobs.xxx\n" +
"\n" +
"'name' is required\n\n",
},
Support multiple paths for diagnostics (#1616) ## Changes Some diagnostics can have multiple paths associated with them. For instance, ensuring that unique resource keys are used across all resources. This PR extends `diag.Diagnostic` to accept multiple paths. This PR is symmetrical to https://github.com/databricks/cli/pull/1610/files ## Tests Unit tests
2024-07-25 15:16:27 +00:00
{
name: "error with multiple paths",
diags: diag.Diagnostics{
{
Severity: diag.Error,
Detail: "'name' is required",
Summary: "failed to load xxx",
Paths: []dyn.Path{
dyn.MustPathFromString("resources.jobs.xxx"),
dyn.MustPathFromString("resources.jobs.yyy"),
dyn.MustPathFromString("resources.jobs.zzz"),
},
},
},
expected: "Error: failed to load xxx\n" +
" at resources.jobs.xxx\n" +
" resources.jobs.yyy\n" +
" resources.jobs.zzz\n" +
"\n" +
"'name' is required\n\n",
},
Add validation for files with a `.(resource-name).yml` extension (#1780) ## Changes We want to encourage a pattern of specifying only a single resource in a YAML file when the `.(resource-type).yml` extension is used (for example, `.job.yml`). This convention could allow us to bijectively map a resource YAML file to its corresponding resource in the Databricks workspace. This PR: 1. Emits a recommendation diagnostic when we detect this convention is being violated. We can promote this to a warning when we want to encourage this pattern more strongly. 2. Visualises the recommendation diagnostics in the `bundle validate` command. **NOTE:** While this PR also shows the recommendation for `.yaml` files, we do not encourage users to use this extension. We only support it here since it's part of the YAML standard and some existing users might already be using `.yaml`. ## Tests Unit tests and manually. Here's what an example output looks like: ``` Recommendation: define a single job in a file with the .job.yml extension. at resources.jobs.bar resources.jobs.foo in foo.job.yml:13:7 foo.job.yml:5:7 The following resources are defined or configured in this file: - bar (job) - foo (job) ``` --------- Co-authored-by: Lennart Kats (databricks) <lennart.kats@databricks.com>
2024-10-07 09:16:20 +00:00
{
name: "recommendation with multiple paths and locations",
diags: diag.Diagnostics{
{
Severity: diag.Recommendation,
Summary: "summary",
Detail: "detail",
Paths: []dyn.Path{
dyn.MustPathFromString("resources.jobs.xxx"),
dyn.MustPathFromString("resources.jobs.yyy"),
},
Locations: []dyn.Location{
{File: "foo.yaml", Line: 1, Column: 2},
{File: "bar.yaml", Line: 3, Column: 4},
},
},
},
expected: "Recommendation: summary\n" +
" at resources.jobs.xxx\n" +
" resources.jobs.yyy\n" +
" in foo.yaml:1:2\n" +
" bar.yaml:3:4\n\n" +
"detail\n\n",
},
Improve `bundle validate` output (#1532) ## Changes This combination of changes allows pretty-printing errors happening during the "load" and "init" phases, including their locations. Move to render code into a separate module dedicated to rendering `diag.Diagnostics` in a human-readable format. This will be used for the `bundle deploy` command. Preserve the "bundle" value if an error occurs in mutators. Rewrite the go templates to handle the case when the bundle isn't yet loaded if an error occurs during loading, that is possible now. Improve rendering for errors and warnings: - don't render empty locations - render "details" for errors if they exist Add `root.ErrAlreadyPrinted` indicating that the error was already printed, and the CLI entry point shouldn't print it again. ## Tests Add tests for output, that are especially handy to detect extra newlines
2024-07-01 09:01:10 +00:00
}
for _, tc := range testCases {
t.Run(tc.name, func(t *testing.T) {
writer := &bytes.Buffer{}
Add a textual bundle summary command
2024-08-24 18:48:36 +00:00
err := renderDiagnosticsOnly(writer, bundle, tc.diags)
Improve `bundle validate` output (#1532) ## Changes This combination of changes allows pretty-printing errors happening during the "load" and "init" phases, including their locations. Move to render code into a separate module dedicated to rendering `diag.Diagnostics` in a human-readable format. This will be used for the `bundle deploy` command. Preserve the "bundle" value if an error occurs in mutators. Rewrite the go templates to handle the case when the bundle isn't yet loaded if an error occurs during loading, that is possible now. Improve rendering for errors and warnings: - don't render empty locations - render "details" for errors if they exist Add `root.ErrAlreadyPrinted` indicating that the error was already printed, and the CLI entry point shouldn't print it again. ## Tests Add tests for output, that are especially handy to detect extra newlines
2024-07-01 09:01:10 +00:00
require.NoError(t, err)
Add a textual bundle summary command
2024-08-24 18:48:36 +00:00
require.Equal(t, tc.expected, writer.String())
Improve `bundle validate` output (#1532) ## Changes This combination of changes allows pretty-printing errors happening during the "load" and "init" phases, including their locations. Move to render code into a separate module dedicated to rendering `diag.Diagnostics` in a human-readable format. This will be used for the `bundle deploy` command. Preserve the "bundle" value if an error occurs in mutators. Rewrite the go templates to handle the case when the bundle isn't yet loaded if an error occurs during loading, that is possible now. Improve rendering for errors and warnings: - don't render empty locations - render "details" for errors if they exist Add `root.ErrAlreadyPrinted` indicating that the error was already printed, and the CLI entry point shouldn't print it again. ## Tests Add tests for output, that are especially handy to detect extra newlines
2024-07-01 09:01:10 +00:00
})
}
}
func TestRenderSummaryTemplate_nilBundle(t *testing.T) {
writer := &bytes.Buffer{}
Add a textual bundle summary command
2024-08-24 18:48:36 +00:00
err := renderSummaryHeaderTemplate(writer, nil)
Improve `bundle validate` output (#1532) ## Changes This combination of changes allows pretty-printing errors happening during the "load" and "init" phases, including their locations. Move to render code into a separate module dedicated to rendering `diag.Diagnostics` in a human-readable format. This will be used for the `bundle deploy` command. Preserve the "bundle" value if an error occurs in mutators. Rewrite the go templates to handle the case when the bundle isn't yet loaded if an error occurs during loading, that is possible now. Improve rendering for errors and warnings: - don't render empty locations - render "details" for errors if they exist Add `root.ErrAlreadyPrinted` indicating that the error was already printed, and the CLI entry point shouldn't print it again. ## Tests Add tests for output, that are especially handy to detect extra newlines
2024-07-01 09:01:10 +00:00
require.NoError(t, err)
Add a textual bundle summary command
2024-08-24 18:48:36 +00:00
io.WriteString(writer, buildTrailer(nil))
require.Equal(t, "Validation OK!\n", writer.String())
}
func TestRenderSummary(t *testing.T) {
ctx := context.Background()
// Create a mock bundle with various resources
b := &bundle.Bundle{
Config: config.Root{
Bundle: config.Bundle{
Name: "test-bundle",
Target: "test-target",
},
Workspace: config.Workspace{
Host: "https://mycompany.databricks.com/",
},
Resources: config.Resources{
Jobs: map[string]*resources.Job{
"job1": {
ID: "1",
URL: "https://url1",
JobSettings: &jobs.JobSettings{Name: "job1-name"},
},
"job2": {
ID: "2",
URL: "https://url2",
JobSettings: &jobs.JobSettings{Name: "job2-name"},
},
},
Pipelines: map[string]*resources.Pipeline{
"pipeline2": {
ID: "4",
// no URL
PipelineSpec: &pipelines.PipelineSpec{Name: "pipeline2-name"},
},
"pipeline1": {
ID: "3",
URL: "https://url3",
PipelineSpec: &pipelines.PipelineSpec{Name: "pipeline1-name"},
},
},
Schemas: map[string]*resources.Schema{
"schema1": {
ID: "catalog.schema",
CreateSchema: &catalog.CreateSchema{
Name: "schema",
},
// no URL
},
},
},
},
}
writer := &bytes.Buffer{}
err := RenderSummary(ctx, writer, b)
require.NoError(t, err)
expectedSummary := `Name: test-bundle
Target: test-target
Workspace:
Host: https://mycompany.databricks.com/
Resources:
Jobs:
job1:
Name: job1-name
URL: https://url1
job2:
Name: job2-name
URL: https://url2
Pipelines:
pipeline1:
Name: pipeline1-name
URL: https://url3
pipeline2:
Name: pipeline2-name
URL: (not deployed)
Schemas:
schema1:
Name: schema
URL: (not deployed)
`
require.Equal(t, expectedSummary, writer.String())
Improve `bundle validate` output (#1532) ## Changes This combination of changes allows pretty-printing errors happening during the "load" and "init" phases, including their locations. Move to render code into a separate module dedicated to rendering `diag.Diagnostics` in a human-readable format. This will be used for the `bundle deploy` command. Preserve the "bundle" value if an error occurs in mutators. Rewrite the go templates to handle the case when the bundle isn't yet loaded if an error occurs during loading, that is possible now. Improve rendering for errors and warnings: - don't render empty locations - render "details" for errors if they exist Add `root.ErrAlreadyPrinted` indicating that the error was already printed, and the CLI entry point shouldn't print it again. ## Tests Add tests for output, that are especially handy to detect extra newlines
2024-07-01 09:01:10 +00:00
}