perf(git): harden large repository maintenance and fetch

[forhappy]

Current implementation head

The PR branch is 68ddf2fc (docs(plan): record visibility bitmap repair). The
code fix is 01d588ea; the latest commit records its failure, invariant, and
released-shape proof in the large-repository roadmap.

Summary

This PR hardens Crab for Kubernetes-scale history and sustained large-team Git
activity on object storage. It combines geometric pack maintenance, a
generation-bound object catalog, bitmap-native visibility planning, split
commit-graph metadata, delta-preserving response assembly, bounded maintenance,
cache-safe generated packs, crash recovery, and a real Kubernetes/RustFS
qualification harness.

The design borrows Git/GitHub’s large-repository ownership model—MIDX-like
object identity, multi-pack reachability bitmaps, split commit graphs,
geometric maintenance, and immutable response artifacts—without copying Git
implementation source or requiring a Git data server.

Latest correctness fix

The first released-shape run after active-marker recovery (32914574732) found
a real shared-state bug in the response-loss lifecycle:

• GitVisibilityIndex::apply_edit grows one shared OID dictionary and resized
  current-ref and incremental-history bitmaps.
• It did not resize bitmap closures retained in transitions for other refs.
• A later validation therefore rejected a valid index with visibility closure bitmap length does not match its dictionary.

01d588ea fixes the invariant by resizing every retained transition bitmap
whenever the dictionary grows. The regression
unrelated_ref_transition_bitmaps_follow_dictionary_growth exercises a dense
transition on one ref, grows the dictionary through an unrelated ref, and
requires catalog identity binding/validation to succeed.

Qualification proof

The fixed code head passed the complete Git protocol v2 qualification
workflow:

• protocol unit and transcript tests;
• released-shape RustFS real-Git lifecycle;
• concurrent-push, response-loss, and crash-recovery lifecycle;
• Git 2.30, 2.40, 2.45, and current compatibility jobs.

The final-head large-repository RustFS qualification workflow is green; its Kubernetes replay job is manual and skipped in pull-request CI, while the qualification-report contract passed. The earlier full Kubernetes replay remains historical evidence. The final head also passed the CI,
architecture,
replica,
cache,
GC, and
NFS gates.

The current Kubernetes full-profile report is
local-k8s-final-04655f3b-1000-20260825: Kubernetes revision
b3bc2ac58fa173967f27ade80f28cc5015b8c1c3, isolated external RustFS,
status=ok, profile=full, replay_count=1000, 22/22 harness checks passed,
1,001 pushes, matching refs and clone tips, full/incremental fsck success, and
1,000 sampled byte-identical objects. Its fingerprint is
7d97627cf1f4de8b87679dea53d99916df42c3152dc765399d4494c43af09624.
That report predates the final handoff fixes and remains historical performance
evidence; the released-shape workflow above is the current correctness proof.

Local validation

• cargo test -p crab-metadata --locked --features storage,remote-index: 240
  passed, 1 ignored, 0 failed.
• Focused transition-bitmap regression: passed.
• Crab upload-pack admission and journal-recovery regressions: passed.
• cargo fmt --all -- --check and git diff --check: passed.

Remaining production-readiness gates

These are explicit evidence gaps, not hidden claims:

1. Repeatable current-binary full-profile runs and a 1,000-push growth/latency
   comparison across multiple runs.
2. 10,000 deterministic Kubernetes ancestry pairs and depth-1/10/100/1,000
   shallow differential proof.
3. Full response-pack SLO proof, including deep/cold clone cost and sustained
   cache fanout; the Kubernetes response pack is roughly 1.2 GB.
4. The complete 10,000-push maintenance/interruption matrix, concurrent
   push/fetch, throttling, cache-server fanout, and owner-failover evidence.
5. Supported-provider range/retry/interruption compatibility, sustained canary,
   and rollout evidence.
6. The owner report’s repair_required state must be explained and cleared
   before production readiness; bucket-registry discovery is incomplete and
   destructive bucket GC remains disabled.
7. A persisted compact catalog sidecar or lazy ordinal lookup path is still
   needed to remove the remaining O(N) visibility dictionary load per
   upload-pack process.

The detailed phased execution, acceptance criteria, evidence table, and STOP
conditions are maintained in
plans/001-large-repository-scale-roadmap.md.
The PR remains draft until these operational gates are either proven or
explicitly accepted as follow-up release criteria.