databricks-cli/libs/sync
Pieter Noordhuis 16bb224108
Add directory tracking to sync (#425)
## Changes

This change replaces usage of the `repofiles` package with the `filer`
package to consolidate WSFS code paths.

The `repofiles` package implemented the following behavior. If a file at
`foo/bar.txt` was created and removed, the directory `foo` was kept
around because we do not perform directory tracking. If subsequently, a
file at `foo` was created, it resulted in an `fs.ErrExist` because it is
impossible to overwrite a directory. It would then perform a recursive
delete of the path if this happened and retry the file write.

To make this use case work without resorting to a recursive delete on
conflict, we need to implement directory tracking as part of sync. The
approach in this commit is as follows:

1. Maintain set of directories needed for current set of files. Compare
to previous set of files. This results in mkdir of added directories and
rmdir of removed directories.
2. Creation of new directories should happen prior to writing files.
Otherwise, many file writes may race to create the same parent
directories, resulting in additional API calls. Removal of existing
directories should happen after removing files.
3. Making new directories can be deduped across common prefixes where
only the longest prefix is created recursively.
4. Removing existing directories must happen sequentially, starting with
the longest prefix.
5. Removal of directories is a best effort. It fails only if the
directory is not empty, and if this happens we know something placed a
file or directory manually, outside of sync.

## Tests

* Existing integration tests pass (modified where it used to assert
directories weren't cleaned up)
* New integration test to confirm the inability to remove a directory
doesn't fail the sync run
2023-06-12 11:44:00 +00:00
..
diff.go Add directory tracking to sync (#425) 2023-06-12 11:44:00 +00:00
diff_test.go Add directory tracking to sync (#425) 2023-06-12 11:44:00 +00:00
dirset.go Add directory tracking to sync (#425) 2023-06-12 11:44:00 +00:00
dirset_test.go Add directory tracking to sync (#425) 2023-06-12 11:44:00 +00:00
event.go Drain sync event channel before returning (#253) 2023-03-16 17:48:17 +01:00
event_test.go Add optional JSON output for sync command (#230) 2023-03-08 10:27:19 +01:00
path.go Rename bricks -> databricks (#389) 2023-05-16 18:35:39 +02:00
path_test.go Update to Go SDK v0.8.0 (#351) 2023-04-21 10:30:20 +02:00
snapshot.go Add directory tracking to sync (#425) 2023-06-12 11:44:00 +00:00
snapshot_test.go Add directory tracking to sync (#425) 2023-06-12 11:44:00 +00:00
sync.go Add directory tracking to sync (#425) 2023-06-12 11:44:00 +00:00
watchdog.go Add directory tracking to sync (#425) 2023-06-12 11:44:00 +00:00