af0114a5a6
Implement readahead cache for Workspace API calls ( #1582 )
...
## Changes
The reason this readahead cache exists is that we frequently need to
recursively find all files in the bundle root directory, especially for
sync include and exclude processing. By caching the response for every
file/directory and frontloading the latency cost of these calls, we
significantly improve performance and eliminate redundant operations.
## Tests
* [ ] Working on unit tests
2024-07-18 09:45:10 +00:00
869576e144
Move bespoke status call to main workspace files filer ( #1570 )
...
## Changes
This consolidates the two separate status calls into one.
The extension-aware filer now doesn't need the direct API client anymore
and fully relies on the underlying filer.
## Tests
* Unit tests.
* Ran the filer integration tests manually.
2024-07-05 11:32:29 +00:00
8957f1e7cf
Return `fs.ModeDir` for Git folders in the workspace ( #1521 )
...
## Changes
Not doing this meant file system traversal ended upon reaching a Git
folder. By marking these objects as a directory globbing traverses into
these folders as well.
## Tests
Added a unit test for coverage.
2024-06-24 10:15:13 +00:00
1752e29885
Update Go SDK to v0.19.0 ( #729 )
...
## Changes
* Update Go SDK to v0.19.0
* Update commands per OpenAPI spec from Go SDK
* Incorporate `client.Do()` signature change to include a (nil) header
map
* Update `workspace.WorkspaceService` mock with permissions methods
* Skip `files` service in codegen; already implemented under the `fs`
command
## Tests
Unit and integration tests pass.
2023-09-05 09:43:57 +00:00
6e708da6fc
Upgraded Go version to 1.21 ( #664 )
...
## Changes
Upgraded Go version to 1.21
Upgraded to use `slices` and `slog` from core instead of experimental.
Still use `exp/maps` as our code relies on `maps.Keys` which is not part
of core package and therefore refactoring required.
### Tests
Integration tests passed
```
[DEBUG] Test execution command: /opt/homebrew/opt/go@1.21/bin/go test ./... -json -timeout 1h -run ^TestAcc
[DEBUG] Test execution directory: /Users/andrew.nester/cli
2023/08/15 13:20:51 [INFO] ✅ TestAccAlertsCreateErrWhenNoArguments (2.150s)
2023/08/15 13:20:52 [INFO] ✅ TestAccApiGet (0.580s)
2023/08/15 13:20:53 [INFO] ✅ TestAccClustersList (0.900s)
2023/08/15 13:20:54 [INFO] ✅ TestAccClustersGet (0.870s)
2023/08/15 13:21:06 [INFO] ✅ TestAccFilerWorkspaceFilesReadWrite (11.980s)
2023/08/15 13:21:13 [INFO] ✅ TestAccFilerWorkspaceFilesReadDir (7.060s)
2023/08/15 13:21:25 [INFO] ✅ TestAccFilerDbfsReadWrite (12.810s)
2023/08/15 13:21:33 [INFO] ✅ TestAccFilerDbfsReadDir (7.380s)
2023/08/15 13:21:41 [INFO] ✅ TestAccFilerWorkspaceNotebookConflict (7.760s)
2023/08/15 13:21:49 [INFO] ✅ TestAccFilerWorkspaceNotebookWithOverwriteFlag (8.660s)
2023/08/15 13:21:49 [INFO] ✅ TestAccFilerLocalReadWrite (0.020s)
2023/08/15 13:21:49 [INFO] ✅ TestAccFilerLocalReadDir (0.010s)
2023/08/15 13:21:52 [INFO] ✅ TestAccFsCatForDbfs (3.190s)
2023/08/15 13:21:53 [INFO] ✅ TestAccFsCatForDbfsOnNonExistentFile (0.890s)
2023/08/15 13:21:54 [INFO] ✅ TestAccFsCatForDbfsInvalidScheme (0.600s)
2023/08/15 13:21:57 [INFO] ✅ TestAccFsCatDoesNotSupportOutputModeJson (2.960s)
2023/08/15 13:22:28 [INFO] ✅ TestAccFsCpDir (31.480s)
2023/08/15 13:22:43 [INFO] ✅ TestAccFsCpFileToFile (14.530s)
2023/08/15 13:22:58 [INFO] ✅ TestAccFsCpFileToDir (14.610s)
2023/08/15 13:23:29 [INFO] ✅ TestAccFsCpDirToDirFileNotOverwritten (31.810s)
2023/08/15 13:23:47 [INFO] ✅ TestAccFsCpFileToDirFileNotOverwritten (17.500s)
2023/08/15 13:24:04 [INFO] ✅ TestAccFsCpFileToFileFileNotOverwritten (17.260s)
2023/08/15 13:24:37 [INFO] ✅ TestAccFsCpDirToDirWithOverwriteFlag (32.690s)
2023/08/15 13:24:56 [INFO] ✅ TestAccFsCpFileToFileWithOverwriteFlag (19.290s)
2023/08/15 13:25:15 [INFO] ✅ TestAccFsCpFileToDirWithOverwriteFlag (19.230s)
2023/08/15 13:25:17 [INFO] ✅ TestAccFsCpErrorsWhenSourceIsDirWithoutRecursiveFlag (2.010s)
2023/08/15 13:25:18 [INFO] ✅ TestAccFsCpErrorsOnInvalidScheme (0.610s)
2023/08/15 13:25:33 [INFO] ✅ TestAccFsCpSourceIsDirectoryButTargetIsFile (14.900s)
2023/08/15 13:25:37 [INFO] ✅ TestAccFsLsForDbfs (3.770s)
2023/08/15 13:25:41 [INFO] ✅ TestAccFsLsForDbfsWithAbsolutePaths (4.160s)
2023/08/15 13:25:44 [INFO] ✅ TestAccFsLsForDbfsOnFile (2.990s)
2023/08/15 13:25:46 [INFO] ✅ TestAccFsLsForDbfsOnEmptyDir (1.870s)
2023/08/15 13:25:46 [INFO] ✅ TestAccFsLsForDbfsForNonexistingDir (0.850s)
2023/08/15 13:25:47 [INFO] ✅ TestAccFsLsWithoutScheme (0.560s)
2023/08/15 13:25:49 [INFO] ✅ TestAccFsMkdirCreatesDirectory (2.310s)
2023/08/15 13:25:52 [INFO] ✅ TestAccFsMkdirCreatesMultipleDirectories (2.920s)
2023/08/15 13:25:55 [INFO] ✅ TestAccFsMkdirWhenDirectoryAlreadyExists (2.320s)
2023/08/15 13:25:57 [INFO] ✅ TestAccFsMkdirWhenFileExistsAtPath (2.820s)
2023/08/15 13:26:01 [INFO] ✅ TestAccFsRmForFile (4.030s)
2023/08/15 13:26:05 [INFO] ✅ TestAccFsRmForEmptyDirectory (3.530s)
2023/08/15 13:26:08 [INFO] ✅ TestAccFsRmForNonEmptyDirectory (3.190s)
2023/08/15 13:26:09 [INFO] ✅ TestAccFsRmForNonExistentFile (0.830s)
2023/08/15 13:26:13 [INFO] ✅ TestAccFsRmForNonEmptyDirectoryWithRecursiveFlag (3.580s)
2023/08/15 13:26:13 [INFO] ✅ TestAccGitClone (0.800s)
2023/08/15 13:26:14 [INFO] ✅ TestAccGitCloneWithOnlyRepoNameOnAlternateBranch (0.790s)
2023/08/15 13:26:15 [INFO] ✅ TestAccGitCloneErrorsWhenRepositoryDoesNotExist (0.540s)
2023/08/15 13:26:23 [INFO] ✅ TestAccLock (8.630s)
2023/08/15 13:26:27 [INFO] ✅ TestAccLockUnlockWithoutAllowsLockFileNotExist (3.490s)
2023/08/15 13:26:30 [INFO] ✅ TestAccLockUnlockWithAllowsLockFileNotExist (3.130s)
2023/08/15 13:26:39 [INFO] ✅ TestAccSyncFullFileSync (9.370s)
2023/08/15 13:26:50 [INFO] ✅ TestAccSyncIncrementalFileSync (10.390s)
2023/08/15 13:27:00 [INFO] ✅ TestAccSyncNestedFolderSync (10.680s)
2023/08/15 13:27:11 [INFO] ✅ TestAccSyncNestedFolderDoesntFailOnNonEmptyDirectory (10.970s)
2023/08/15 13:27:22 [INFO] ✅ TestAccSyncNestedSpacePlusAndHashAreEscapedSync (10.930s)
2023/08/15 13:27:29 [INFO] ✅ TestAccSyncIncrementalFileOverwritesFolder (7.020s)
2023/08/15 13:27:37 [INFO] ✅ TestAccSyncIncrementalSyncPythonNotebookToFile (7.380s)
2023/08/15 13:27:43 [INFO] ✅ TestAccSyncIncrementalSyncFileToPythonNotebook (6.050s)
2023/08/15 13:27:48 [INFO] ✅ TestAccSyncIncrementalSyncPythonNotebookDelete (5.390s)
2023/08/15 13:27:51 [INFO] ✅ TestAccSyncEnsureRemotePathIsUsableIfRepoDoesntExist (2.570s)
2023/08/15 13:27:56 [INFO] ✅ TestAccSyncEnsureRemotePathIsUsableIfRepoExists (5.540s)
2023/08/15 13:27:58 [INFO] ✅ TestAccSyncEnsureRemotePathIsUsableInWorkspace (1.840s)
2023/08/15 13:27:59 [INFO] ✅ TestAccWorkspaceList (0.790s)
2023/08/15 13:28:08 [INFO] ✅ TestAccExportDir (8.860s)
2023/08/15 13:28:11 [INFO] ✅ TestAccExportDirDoesNotOverwrite (3.090s)
2023/08/15 13:28:14 [INFO] ✅ TestAccExportDirWithOverwriteFlag (3.500s)
2023/08/15 13:28:23 [INFO] ✅ TestAccImportDir (8.330s)
2023/08/15 13:28:34 [INFO] ✅ TestAccImportDirDoesNotOverwrite (10.970s)
2023/08/15 13:28:44 [INFO] ✅ TestAccImportDirWithOverwriteFlag (10.130s)
2023/08/15 13:28:44 [INFO] ✅ 68/68 passed, 0 failed, 3 skipped
```
2023-08-15 13:50:40 +00:00
30efe91c6d
Make local files default for fs commands ( #506 )
...
## Changes
<!-- Summary of your changes that are easy to understand -->
## Tests
<!-- How is this tested? -->
2023-06-23 16:07:09 +02:00
d0e9953ad9
Use direct download for workspace filer read ( #514 )
...
## Changes
Use the download method from the SDK in the read method for the WSFS
implementation of the filer interface.
Closes #452 .
## Tests
Tested by existing integration tests
2023-06-23 15:17:39 +02:00
d38649088c
Add workspace import-dir command ( #456 )
...
## Tests
Testing using integration tests and manually
2023-06-12 21:03:46 +02:00
960ce2e18e
Add implementation of filer.Filer for local filesystem ( #460 )
...
## Changes
Local file reads on Windows require the file handle to be closed after
using it. This commit includes an interface change to return an
`io.ReadCloser` from `Read` to accommodate this.
## Tests
The existing integration tests for the filer interface all pass.
2023-06-12 15:53:58 +02:00
4818541062
Add workspace export-dir command ( #449 )
...
## Changes
This PR:
1. Adds the export-dir command
2. Changes filer.Read to return an error if a user tries to read a
directory
3. Adds returning internal file structures from filer.Stat().Sys()
## Tests
Integration tests and manually
2023-06-08 18:15:12 +02:00
be10ff9a75
Include recursive deletion in filer interface ( #442 )
...
## Changes
This captures the recursive deletion of a directory tree in the filer interface.
Prompted by #433 .
## Tests
Integration tests pass (ran the filer ones on AWS and Azure).
2023-06-06 06:27:47 +00:00
91097856b5
Add check for path is a directory in filer.ReadDir ( #426 )
...
## Tests
Integration tests
2023-06-02 12:28:35 +02:00
2b56af6016
Add Stat function to filer.Filer interface ( #421 )
...
## Changes
TSIA
## Tests
New integration test passes.
2023-06-01 20:23:22 +02:00
42cd8daee0
Make filer.Filer return fs.DirEntry from ReadDir ( #415 )
...
## Changes
This allows for compatibility with the stdlib functions in io/fs.
## Tests
Integration tests pass.
2023-05-31 14:22:26 +02:00
27df4e765c
Implement DBFS filer ( #139 )
...
Adds a DBFS implementation of the `filer.Filer` interface.
The integration tests are reused between the workspace filesystem and
DBFS implementations to ensure identical behavior.
2023-05-31 13:24:20 +02:00
92cb52041d
Add Mkdir and ReadDir functions to filer.Filer interface ( #414 )
...
## Changes
This cherry-picks the filer changes from #408 .
## Tests
Manually ran integration tests.
2023-05-31 11:11:17 +02:00
ae09eb02d5
Path escape file path in filer interface ( #254 )
2023-03-17 17:42:35 +01:00
65b3f998ba
Escape URL in filer ( #236 )
...
Also see #228 .
2023-03-08 14:27:05 +01:00
414ea4f891
Bump databricks-sdk-go to 0.3.2 ( #215 )
2023-02-20 16:00:20 +01:00
a59136f77f
Use []byte for files in workspace ( #162 )
2023-01-05 12:03:31 +01:00
32a37c1b83
Use filer.Filer in bundle/deployer/locker ( #136 )
...
Summary:
* All remote path arguments for deployer and locker are now relative to
root specified at initialization
* The workspace client is now a struct field so it doesn't have to be
passed around
2022-12-15 17:16:07 +01:00
4e834857e6
Extract filer path handling into separate type ( #138 )
...
This makes it reusable for the DBFS filer.
2022-12-14 23:41:37 +01:00
12aae35519
Abstract over file handling with WSFS or DBFS through filer interface ( #135 )
2022-12-14 15:37:14 +01:00
Pieter Noordhuis
Andrew Nester
shreyas-goenka