Harden large-file add/push pipeline: resumable multipart, streaming smudge/LFS, bounded memory, concurrent packs - #70
Open
forhappy wants to merge 8 commits into
Open
Conversation
Add MultipartJournal trait and put_multipart_file_resumable_retry: providers with explicit upload-id APIs (S3/GCS/Azure/memory) now record completed parts so cancelled pushes resume from the last good part on the next run. Hard errors still clean up their session; cancellation leaves it alive for fsck-grace reclamation. Provider builders capture the low-level MultipartStore handle at construction (same pattern as the signer handle) and thread it through auth-store and product store wrappers.
Open the staging multipart registry at push start and pass it through the resumable upload path for xorbs and packs. Cancelled or killed pushes now record completed parts, so the next run uploads only the missing suffix; resumed uploads increment the existing multipart_resumed_uploads metric. fsck gains real abandoned-upload detection and repair against that registry (previously a stub returning an empty list). The registry's schema generalizes xorb_hash to payload_hash so packs share the same journal, and begin() refuses to clobber a concurrent uploader's row.
crab lfs push / pre-push uploaded each object via a full in-memory read (std::fs::read) before a single PUT — an OOM risk on multi-GiB objects. Route through LfsObjectStore::put_stream, which streams fixed-size parts with inline SHA-256 verification and aborts the multipart on mismatch. The now-unused whole-object reader is deleted; the explicit HEAD-skip stays so missing local objects fail before any upload session opens.
The smudge arm accumulated every incoming blob in memory (1 MiB Vec growing to full file size, plus a second copy in the result). Pointers fit one packet, so classification now works from a bounded first-packet peek; raw multi-GiB content relays packet-to-packet with no buffering. Inline crab reconstruction spills through a temp file (reconstruct_to_ path already existed) and streams out in 256 KiB frames — preserving the old fall-back-to-pointer behavior when reconstruction fails before any response bytes are written. read_content_until_flush and the in-memory smudge_content paths are deleted.
read_prepared_staged_chunks_batch spawned one blocking task per distinct prepared xorb, each loading up to 256 MiB into RAM — RSS on push scaled with add size. Acquire from a 4-permit semaphore before spawning; the join loop keeps result order stable.
Packs uploaded strictly sequentially while xorbs/shards already ran concurrently — large-history pushes serialized on the slowest stage. Extract per-pack work into upload_single_pack (install, resumable body upload, idx/rev evidence, CAS metadata, origin receipt; all keyed by content hash so iterations are order-independent) and drive it through a 2-permit semaphore. Results re-order by input index so downstream locator publication stays deterministic; first error drains in-flight siblings, matching the xorb pipeline's semantics.
Multipart completes carry no provider-neutral integrity guarantee (S3 ETags are md5-of-part-etags), so fresh multipart uploads now read back and blake3-verify the stored body before the push publishes pointers over it. Single-PUT uploads ride on provider-enforced checksums and stay verification-free — a per-xorb HEAD doubled request volume on chunk-heavy pushes for marginal value. A mismatch fails the push loudly; re-push recovers via dedup proofs or overwrite repair.
- Move crab add's git-index publication into spawn_blocking: gix ODB writes plus the index flock/commit are synchronous and stalled async workers for seconds on large batches. The writer now consumes owned entry snapshots so the blocking closure stays 'static; a panicked writer maps to IndexMutationUncertain, preserving staged rows as the retry source. - Delete dead clean::DEFAULT_FASTPATH_MIN_SIZE (64 MiB) — the live default is config.rs's 1 MiB constant. - Fix stale architecture docs pointing at a nonexistent crab/src/storage/multipart_resume.rs; document the wired journal, resume behavior, and multipart read-back verification instead.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Hardens the large-file path end-to-end (add → commit → push → checkout) across 8 commits:
Resumable multipart uploads (P0)
MultipartJournaltrait +Store::put_multipart_file_resumable_retry: S3/GCS/Azure explicit upload-id APIs drive part-level resume. Cancelled or killed pushes record completed parts in.crab/staging/multipart.dband re-upload only the missing suffix next run.multipart_resumed_uploadsmetric increments on real resumes.crab fsckgains real abandoned-upload detection/repair against the journal (was a stub returning an empty list).Memory bounds (P0/P1)
crab lfs pushstreams viaput_streamwith inline SHA-256 verification — no more whole-object reads (OOM risk on multi-GiB objects).Throughput & integrity
Hygiene
crab add's git-index publication moved off async workers (spawn_blocking, panicked-writer →IndexMutationUncertainpreserving staged rows).clean::DEFAULT_FASTPATH_MIN_SIZEremoved; stale architecture docs now describe the wired journal/resume/verification behavior.Test plan
crablib suite: 3,687 passed / 0 failed on rebased basee2e_add_commit_push,add_ship_contract,prop_push_batch, filter-process suite (27), pack/upload suites all greenMultipartStorefake)Notes for reviewers
.crab/staging/already exists — clean checkouts grow no new DB.store.getstill buffers; needs an LFS-store writer API).