By keeping a serialized copier state on disk, we could implement a resumable zarr copier. That would be really handy for when a reboot is needed during a long copy job from the local fs to a compute cluster, for example. There are tools that can do this, but none are zarr-specific and so they can't utilize the structure of the archives in the way we can to speed things up (nobody wants a filelist with 35M entries)
By keeping a serialized copier state on disk, we could implement a resumable zarr copier. That would be really handy for when a reboot is needed during a long copy job from the local fs to a compute cluster, for example. There are tools that can do this, but none are zarr-specific and so they can't utilize the structure of the archives in the way we can to speed things up (nobody wants a filelist with 35M entries)