steve/databricks-cli
steve
/
databricks-cli
mirror of https://github.com/databricks/cli.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
901 Commits 97 Branches 105 Tags 59 MiB
Commit Graph

4 Commits

Author SHA1 Message Date
Miles Yucht b65ce75c1f
Use Go SDK Iterators when listing resources with the CLI (#1202) 
## Changes
Currently, when the CLI run a list API call (like list jobs), it uses
the `List*All` methods from the SDK, which list all resources in the
collection. This is very slow for large collections: if you need to list
all jobs from a workspace that has 10,000+ jobs, you'll be waiting for
at least 100 RPCs to complete before seeing any output.

Instead of using List*All() methods, the SDK recently added an iterator
data structure that allows traversing the collection without needing to
completely list it first. New pages are fetched lazily if the next
requested item belongs to the next page. Using the List() methods that
return these iterators, the CLI can proactively print out some of the
response before the complete collection has been fetched.

This involves a pretty major rewrite of the rendering logic in `cmdio`.
The idea there is to define custom rendering logic based on the type of
the provided resource. There are three renderer interfaces:

1. textRenderer: supports printing something in a textual format (i.e.
not JSON, and not templated).
2. jsonRenderer: supports printing something in a pretty-printed JSON
format.
3. templateRenderer: supports printing something using a text template.

There are also three renderer implementations:

1. readerRenderer: supports printing a reader. This only implements the
textRenderer interface.
2. iteratorRenderer: supports printing a `listing.Iterator` from the Go
SDK. This implements jsonRenderer and templateRenderer, buffering 20
resources at a time before writing them to the output.
3. defaultRenderer: supports printing arbitrary resources (the previous
implementation).

Callers will either use `cmdio.Render()` for rendering individual
resources or `io.Reader` or `cmdio.RenderIterator()` for rendering an
iterator. This separate method is needed to safely be able to match on
the type of the iterator, since Go does not allow runtime type matches
on generic types with an existential type parameter.

One other change that needs to happen is to split the templates used for
text representation of list resources into a header template and a row
template. The template is now executed multiple times for List API
calls, but the header should only be printed once. To support this, I
have added `headerTemplate` to `cmdIO`, and I have also changed
`RenderWithTemplate` to include a `headerTemplate` parameter everywhere.

## Tests
- [x] Unit tests for text rendering logic
- [x] Unit test for reflection-based iterator construction.

---------

Co-authored-by: Andrew Nester <andrew.nester@databricks.com>
 2024-02-21 14:16:36 +00:00
shreyas-goenka 76840176e3
Add documentation for positional args in commands generated from the Databricks OpenAPI specification (#1033) 
## Changes
This PR adds documentation for positional arguments in commands that are
generated from the openapi spec.

Note: the changes to `.gitattributes` will be revert / properly fixed in
https://github.com/databricks/cli/pull/1012
 2023-11-30 16:22:23 +00:00
Pieter Noordhuis 0cd3bb072d
Bump Go SDK to v0.26.0 (#1019) 
## Changes

Bump Go SDK to v0.26.0.

Changelog at
https://github.com/databricks/databricks-sdk-go/releases/tag/v0.26.0.

## Tests

Integration tests pass.
 2023-11-29 13:29:31 +00:00
dependabot[bot] c3ced68c60
Bump github.com/databricks/databricks-sdk-go from 0.24.0 to 0.25.0 (#980) 
Bumps
[github.com/databricks/databricks-sdk-go](https://github.com/databricks/databricks-sdk-go)
from 0.24.0 to 0.25.0.
<details>
<summary>Changelog</summary>
<p><em>Sourced from <a
href="https://github.com/databricks/databricks-sdk-go/blob/main/CHANGELOG.md">github.com/databricks/databricks-sdk-go's
changelog</a>.</em></p>
<blockquote>
<h2>0.25.0</h2>
<ul>
<li>Make sure path parameters are first in order in RequiredFields (<a
href="https://redirect.github.com/databricks/databricks-sdk-go/pull/669">#669</a>).</li>
<li>Added Field.IsRequestBodyField method for code generation (<a
href="https://redirect.github.com/databricks/databricks-sdk-go/pull/670">#670</a>).</li>
<li>Added regressions question to the issue template (<a
href="https://redirect.github.com/databricks/databricks-sdk-go/pull/676">#676</a>).</li>
<li>Added telemetry for CI/CD platform to useragent (<a
href="https://redirect.github.com/databricks/databricks-sdk-go/pull/665">#665</a>).</li>
<li>Skiped GCP Integration Tests using Statement Execution API (<a
href="https://redirect.github.com/databricks/databricks-sdk-go/pull/678">#678</a>).</li>
<li>Added more detailed error message on default credentials not found
error (<a
href="https://redirect.github.com/databricks/databricks-sdk-go/pull/679">#679</a>).</li>
<li>Updated SDK to latest OpenAPI Spec (<a
href="https://redirect.github.com/databricks/databricks-sdk-go/pull/685">#685</a>).</li>
</ul>
<p>API Changes:</p>
<ul>
<li>Changed <code>Create</code> method for <a
href="https://pkg.go.dev/github.com/databricks/databricks-sdk-go/service/catalog#FunctionsAPI">w.Functions</a>
and <a
href="https://pkg.go.dev/github.com/databricks/databricks-sdk-go/service/catalog#MetastoresAPI">w.Metastores</a>
workspace-level service with new required argument order.</li>
<li>Changed <code>InputParams</code> field for <a
href="https://pkg.go.dev/github.com/databricks/databricks-sdk-go/service/catalog#CreateFunction">catalog.CreateFunction</a>
and <a
href="https://pkg.go.dev/github.com/databricks/databricks-sdk-go/service/catalog#FunctionInfo">catalog.FunctionInfo</a>
to <a
href="https://pkg.go.dev/github.com/databricks/databricks-sdk-go/service/catalog#FunctionParameterInfos">catalog.FunctionParameterInfos</a>.</li>
<li>Changed <code>Properties</code> field for <a
href="https://pkg.go.dev/github.com/databricks/databricks-sdk-go/service/catalog#CreateFunction">catalog.CreateFunction</a>
and <a
href="https://pkg.go.dev/github.com/databricks/databricks-sdk-go/service/catalog#FunctionInfo">catalog.FunctionInfo</a>
to <code>string</code>.</li>
<li>Changed <code>ReturnParams</code> field for <a
href="https://pkg.go.dev/github.com/databricks/databricks-sdk-go/service/catalog#CreateFunction">catalog.CreateFunction</a>
and <a
href="https://pkg.go.dev/github.com/databricks/databricks-sdk-go/service/catalog#FunctionInfo">catalog.FunctionInfo</a>
to <a
href="https://pkg.go.dev/github.com/databricks/databricks-sdk-go/service/catalog#FunctionParameterInfos">catalog.FunctionParameterInfos</a></li>
<li>Changed <code>StorageRoot</code> field for <a
href="https://pkg.go.dev/github.com/databricks/databricks-sdk-go/service/catalog#CreateMetastore">catalog.CreateMetastore</a>
to no longer be required.</li>
<li>Added <code>SkipValidation</code> field for <a
href="https://pkg.go.dev/github.com/databricks/databricks-sdk-go/service/catalog#UpdateExternalLocation">catalog.UpdateExternalLocation</a>.</li>
<li>Added <code>Libraries</code> field for <a
href="https://pkg.go.dev/github.com/databricks/databricks-sdk-go/service/compute#CreatePolicy">compute.CreatePolicy</a>,
<a
href="https://pkg.go.dev/github.com/databricks/databricks-sdk-go/service/compute#EditPolicy">compute.EditPolicy</a>
and <a
href="https://pkg.go.dev/github.com/databricks/databricks-sdk-go/service/compute#Policy">compute.Policy</a>.</li>
<li>Added <a
href="https://pkg.go.dev/github.com/databricks/databricks-sdk-go/service/compute#InitScriptEventDetails">compute.InitScriptEventDetails</a>.</li>
<li>Added <code>InitScripts</code> field for <a
href="https://pkg.go.dev/github.com/databricks/databricks-sdk-go/service/compute#EventDetails">compute.EventDetails</a>.</li>
<li>Added <code>File</code> field for <a
href="https://pkg.go.dev/github.com/databricks/databricks-sdk-go/service/compute#InitScriptInfo">compute.InitScriptInfo</a>.</li>
<li>Added <code>ZoneId</code> field for <a
href="https://pkg.go.dev/github.com/databricks/databricks-sdk-go/service/compute#InstancePoolGcpAttributes">compute.InstancePoolGcpAttributes</a>.</li>
<li>Added <code>IncludeResolvedValues</code> field for <a
href="https://pkg.go.dev/github.com/databricks/databricks-sdk-go/service/jobs#GetRunRequest">jobs.GetRunRequest</a>.</li>
<li>Added <code>EditMode</code> field for <a
href="https://pkg.go.dev/github.com/databricks/databricks-sdk-go/service/jobs#CreateJob">jobs.CreateJob</a>
and <a
href="https://pkg.go.dev/github.com/databricks/databricks-sdk-go/service/jobs#JobSettings">jobs.JobSettings</a>.</li>
<li>Added <code>NetworkConnectivityConfigId</code> field for <a
href="https://pkg.go.dev/github.com/databricks/databricks-sdk-go/service/provisioning#UpdateWorkspaceRequest">provisioning.UpdateWorkspaceRequest</a>.</li>
<li>Added <code>ContainerLogs</code> and <code>ExtraInfo</code> field
for <a
href="https://pkg.go.dev/github.com/databricks/databricks-sdk-go/service/serving#DeploymentStatus">serving.DeploymentStatus</a>.</li>
<li>Added <a
href="https://pkg.go.dev/github.com/databricks/databricks-sdk-go/service/catalog#CreateFunctionRequest">catalog.CreateFunctionRequest</a>,
<a
href="https://pkg.go.dev/github.com/databricks/databricks-sdk-go/service/catalog#DependencyList">catalog.DependencyList</a>
and <a
href="https://pkg.go.dev/github.com/databricks/databricks-sdk-go/service/catalog#FunctionParameterInfos">catalog.FunctionParameterInfos</a>.</li>
<li>Added <a
href="https://pkg.go.dev/github.com/databricks/databricks-sdk-go/service/compute#InitScriptExecutionDetails">compute.InitScriptExecutionDetails</a>,
<a
href="https://pkg.go.dev/github.com/databricks/databricks-sdk-go/service/compute#InitScriptExecutionDetailsStatus">compute.InitScriptExecutionDetailsStatus</a>,
<a
href="https://pkg.go.dev/github.com/databricks/databricks-sdk-go/service/compute#InitScriptInfoAndExecutionDetails">compute.InitScriptInfoAndExecutionDetails</a>,
<a
href="https://pkg.go.dev/github.com/databricks/databricks-sdk-go/service/compute#LocalFileInfo">compute.LocalFileInfo</a>.</li>
<li>Added <a
href="https://pkg.go.dev/github.com/databricks/databricks-sdk-go/service/jobs#CreateJobEditMode">jobs.CreateJobEditMode</a>
and <a
href="https://pkg.go.dev/github.com/databricks/databricks-sdk-go/service/jobs#JobSettingsEditMode">jobs.JobSettingsEditMode</a>.</li>
<li>Added <code>DeleteApp</code>, <code>GetApp</code>,
<code>GetAppDeploymentStatus</code>, <code>GetApps</code> and
<code>GetEvents</code> method for <a
href="https://pkg.go.dev/github.com/databricks/databricks-sdk-go/service/serving#AppsAPI">w.Apps</a>
workspace-level service.</li>
<li>Added <a
href="https://pkg.go.dev/github.com/databricks/databricks-sdk-go/service/serving#AppEvents">serving.AppEvents</a>,
<a
href="https://pkg.go.dev/github.com/databricks/databricks-sdk-go/service/serving#AppServiceStatus">serving.AppServiceStatus</a>,
<a
href="https://pkg.go.dev/github.com/databricks/databricks-sdk-go/service/serving#DeleteAppResponse">serving.DeleteAppResponse</a>,
<a
href="https://pkg.go.dev/github.com/databricks/databricks-sdk-go/service/serving#GetAppDeploymentStatusRequest">serving.GetAppDeploymentStatusRequest</a>,
<a
href="https://pkg.go.dev/github.com/databricks/databricks-sdk-go/service/serving#GetAppResponse">serving.GetAppResponse</a>,
<a
href="https://pkg.go.dev/github.com/databricks/databricks-sdk-go/service/serving#GetEventsRequest">serving.GetEventsRequest</a>,
<a
href="https://pkg.go.dev/github.com/databricks/databricks-sdk-go/service/serving#ListAppEventsResponse">serving.ListAppEventsResponse</a>
and <a
href="https://pkg.go.dev/github.com/databricks/databricks-sdk-go/service/serving#ListAppsResponse">serving.ListAppsResponse</a>.</li>
<li>Added <a
href="https://pkg.go.dev/github.com/databricks/databricks-sdk-go/service/settings#NetworkConnectivityAPI">a.NetworkConnectivity</a>
account-level service.</li>
<li>Added <a
href="https://pkg.go.dev/github.com/databricks/databricks-sdk-go/service/settings#CreateNetworkConnectivityConfigRequest">settings.CreateNetworkConnectivityConfigRequest</a>,
<a
href="https://pkg.go.dev/github.com/databricks/databricks-sdk-go/service/settings#CreatePrivateEndpointRuleRequest">settings.CreatePrivateEndpointRuleRequest</a>,
<a
href="https://pkg.go.dev/github.com/databricks/databricks-sdk-go/service/settings#CreatePrivateEndpointRuleRequestGroupId">settings.CreatePrivateEndpointRuleRequestGroupId</a>,
<a
href="https://pkg.go.dev/github.com/databricks/databricks-sdk-go/service/settings#DeleteNetworkConnectivityConfigurationRequest">settings.DeleteNetworkConnectivityConfigurationRequest</a>,
<a
href="https://pkg.go.dev/github.com/databricks/databricks-sdk-go/service/settings#DeletePrivateEndpointRuleRequest">settings.DeletePrivateEndpointRuleRequest</a>,
<a
href="https://pkg.go.dev/github.com/databricks/databricks-sdk-go/service/settings#GetNetworkConnectivityConfigurationRequest">settings.GetNetworkConnectivityConfigurationRequest</a>,
<a
href="https://pkg.go.dev/github.com/databricks/databricks-sdk-go/service/settings#GetPrivateEndpointRuleRequest">settings.GetPrivateEndpointRuleRequest</a>,
<a
href="https://pkg.go.dev/github.com/databricks/databricks-sdk-go/service/settings#NccAzurePrivateEndpointRule">settings.NccAzurePrivateEndpointRule</a>,
<a
href="https://pkg.go.dev/github.com/databricks/databricks-sdk-go/service/settings#NccAzurePrivateEndpointRuleConnectionState">settings.NccAzurePrivateEndpointRuleConnectionState</a>,
<a
href="https://pkg.go.dev/github.com/databricks/databricks-sdk-go/service/settings#NccAzurePrivateEndpointRuleGroupId">settings.NccAzurePrivateEndpointRuleGroupId</a>,
<a
href="https://pkg.go.dev/github.com/databricks/databricks-sdk-go/service/settings#NccAzureServiceEndpointRule">settings.NccAzureServiceEndpointRule</a>,
<a
href="https://pkg.go.dev/github.com/databricks/databricks-sdk-go/service/settings#NccEgressConfig">settings.NccEgressConfig</a>,
<a
href="https://pkg.go.dev/github.com/databricks/databricks-sdk-go/service/settings#NccEgressDefaultRules">settings.NccEgressDefaultRules</a>,
<a
href="https://pkg.go.dev/github.com/databricks/databricks-sdk-go/service/settings#NccEgressTargetRules">settings.NccEgressTargetRules</a>,
<a
href="https://pkg.go.dev/github.com/databricks/databricks-sdk-go/service/settings#NetworkConnectivityConfiguration">settings.NetworkConnectivityConfiguration</a>.</li>
<li>Removed <code>Delete</code>, <code>Get</code>, method for <a
href="https://pkg.go.dev/github.com/databricks/databricks-sdk-go/service/serving#AppsAPI">w.Apps</a>
workspace-level service.</li>
<li>Removed <a
href="https://pkg.go.dev/github.com/databricks/databricks-sdk-go/service/jobs#JobSettingsUiState">jobs.JobSettingsUiState</a>
and <a
href="https://pkg.go.dev/github.com/databricks/databricks-sdk-go/service/jobs#CreateJobUiState">jobs.CreateJobUiState</a>.</li>
<li>Removed <a
href="https://pkg.go.dev/github.com/databricks/databricks-sdk-go/service/oauth2#OAuthEnrollmentAPI">a.OAuthEnrollment</a>
account-level service.</li>
<li>Removed <a
href="https://pkg.go.dev/github.com/databricks/databricks-sdk-go/service/oauth2#CreateOAuthEnrollment">oauth2.CreateOAuthEnrollment</a>
and <a
href="https://pkg.go.dev/github.com/databricks/databricks-sdk-go/service/oauth2#OAuthEnrollmentStatus">oauth2.OAuthEnrollmentStatus</a>.</li>
<li>Removed <code>UiState</code> field for <a
href="https://pkg.go.dev/github.com/databricks/databricks-sdk-go/service/jobs#CreateJob">jobs.CreateJob</a>
and <a
href="https://pkg.go.dev/github.com/databricks/databricks-sdk-go/service/jobs#JobSettings">jobs.JobSettings</a>.</li>
</ul>
<p>OpenAPI SHA: e7b127cb07af8dd4d8c61c7cc045c8910cdbb02a, Date:
2023-11-08
Dependency updates:</p>
<ul>
<li>Bump google.golang.org/api from 0.146.0 to 0.150.0 (<a
href="https://redirect.github.com/databricks/databricks-sdk-go/pull/683">#683</a>).</li>
<li>Bump golang.org/x/mod from 0.13.0 to 0.14.0 (<a
href="https://redirect.github.com/databricks/databricks-sdk-go/pull/681">#681</a>).</li>
<li>Bump google.golang.org/grpc from 1.58.2 to 1.58.3 in /examples/slog
(<a
href="https://redirect.github.com/databricks/databricks-sdk-go/pull/672">#672</a>).</li>
<li>Bump google.golang.org/grpc to 1.58.3 in /examples/zerolog (<a
href="https://redirect.github.com/databricks/databricks-sdk-go/pull/684">#684</a>).</li>
<li>Bump golang.org/x/time from 0.3.0 to 0.4.0 (<a
href="https://redirect.github.com/databricks/databricks-sdk-go/pull/680">#680</a>).</li>
</ul>
</blockquote>
</details>
<details>
<summary>Commits</summary>
<ul>
<li><a
href="03b614c890"><code>03b614c</code></a>
Release v0.25.0 (<a
href="https://redirect.github.com/databricks/databricks-sdk-go/issues/687">#687</a>)</li>
<li><a
href="16970dfeb3"><code>16970df</code></a>
Update SDK to latest OpenAPI Spec (<a
href="https://redirect.github.com/databricks/databricks-sdk-go/issues/685">#685</a>)</li>
<li><a
href="712fcfed33"><code>712fcfe</code></a>
Bump golang.org/x/time from 0.3.0 to 0.4.0 (<a
href="https://redirect.github.com/databricks/databricks-sdk-go/issues/680">#680</a>)</li>
<li><a
href="3215e790ed"><code>3215e79</code></a>
Bump google.golang.org/grpc from 1.58.2 to 1.58.3 in /examples/zerolog
(<a
href="https://redirect.github.com/databricks/databricks-sdk-go/issues/684">#684</a>)</li>
<li><a
href="f06dd7330c"><code>f06dd73</code></a>
Bump google.golang.org/grpc from 1.58.2 to 1.58.3 in /examples/slog (<a
href="https://redirect.github.com/databricks/databricks-sdk-go/issues/672">#672</a>)</li>
<li><a
href="2bb6c8de98"><code>2bb6c8d</code></a>
Bump google.golang.org/grpc from 1.57.0 to 1.57.1 in /examples/zerolog
(<a
href="https://redirect.github.com/databricks/databricks-sdk-go/issues/674">#674</a>)</li>
<li><a
href="e045c01326"><code>e045c01</code></a>
Bump golang.org/x/mod from 0.13.0 to 0.14.0 (<a
href="https://redirect.github.com/databricks/databricks-sdk-go/issues/681">#681</a>)</li>
<li><a
href="7e28d29f91"><code>7e28d29</code></a>
Bump google.golang.org/api from 0.146.0 to 0.150.0 (<a
href="https://redirect.github.com/databricks/databricks-sdk-go/issues/683">#683</a>)</li>
<li><a
href="6e2bb1def4"><code>6e2bb1d</code></a>
Add more detailed error message on default credentials not found error
(<a
href="https://redirect.github.com/databricks/databricks-sdk-go/issues/679">#679</a>)</li>
<li><a
href="33c1c8718f"><code>33c1c87</code></a>
Skip GCP Integration Tests using Statement Execution API (<a
href="https://redirect.github.com/databricks/databricks-sdk-go/issues/678">#678</a>)</li>
<li>Additional commits viewable in <a
href="https://github.com/databricks/databricks-sdk-go/compare/v0.24.0...v0.25.0">compare
view</a></li>
</ul>
</details>
<br />


[![Dependabot compatibility
score](https://dependabot-badges.githubapp.com/badges/compatibility_score?dependency-name=github.com/databricks/databricks-sdk-go&package-manager=go_modules&previous-version=0.24.0&new-version=0.25.0)](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores)

Dependabot will resolve any conflicts with this PR as long as you don't
alter it yourself. You can also trigger a rebase manually by commenting
`@dependabot rebase`.

[//]: # (dependabot-automerge-start)
[//]: # (dependabot-automerge-end)

---

<details>
<summary>Dependabot commands and options</summary>
<br />

You can trigger Dependabot actions by commenting on this PR:
- `@dependabot rebase` will rebase this PR
- `@dependabot recreate` will recreate this PR, overwriting any edits
that have been made to it
- `@dependabot merge` will merge this PR after your CI passes on it
- `@dependabot squash and merge` will squash and merge this PR after
your CI passes on it
- `@dependabot cancel merge` will cancel a previously requested merge
and block automerging
- `@dependabot reopen` will reopen this PR if it is closed
- `@dependabot close` will close this PR and stop Dependabot recreating
it. You can achieve the same result by closing it manually
- `@dependabot show <dependency name> ignore conditions` will show all
of the ignore conditions of the specified dependency
- `@dependabot ignore this major version` will close this PR and stop
Dependabot creating any more for this major version (unless you reopen
the PR or upgrade to it yourself)
- `@dependabot ignore this minor version` will close this PR and stop
Dependabot creating any more for this minor version (unless you reopen
the PR or upgrade to it yourself)
- `@dependabot ignore this dependency` will close this PR and stop
Dependabot creating any more for this dependency (unless you reopen the
PR or upgrade to it yourself)


</details>

---------

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Andrew Nester <andrew.nester@databricks.com>
 2023-11-13 15:38:35 +00:00