5.2.2 blob-gap wedge is bounded but banks zero copy progress — the copy cursor never persists during a copy body, and the watchdog's reconnect mints the next cycle's faults

[harper-joseph]

Follow-up to #683 (closed in 5.2.2). Field data from the 4-node Kohls prerender production cluster,
plus a deterministic local reproduction and a fault-free control that together pin the mechanism.

Fix pair in review: harper-pro#701
(per-walk-position watermark banking + replication.blobGapReconnectMs) and
harper#2177 (repair of dangling blob references on
copy re-delivery). One coverage note from the follow-up analysis on #701: the repair heals copy
receivers; stub-damaged references on a node that never receives a base copy (yc0 here — the
topology's root source) have no healing path in the pair, so its outbound copy links latch at those
keys until the records re-render/evict or a source-side change lands.

TL;DR

The blob-gap watchdog works exactly as designed — it fires, it bounds the wedge to blobTimeout.
But the durable copy cursor advances zero keys, so every cycle discards its own progress and
re-walks the same prefix. Bounded ≠ progressing.

Two measured facts drive everything below, and together they rule out every "reduce the fault rate"
mitigation as sufficient:

1. The cursor banks nothing during a copy body even at zero faults. A fault-free control run
   persisted the cursor only after bulk copy complete from — never during the copy.
2. The wedge is not rate-sensitive. One latching fault per watchdog cycle zeroes that cycle.
   Production runs ~588 latching faults per 30 min, but you only need one per 15 minutes.

Root cause — three compounding defects

All line refs replication/replicationConnection.ts @ v5.2.2 unless noted.

D1 — the copy cursor never persists during the copy body

flushDurableCopyCursor() (:2413) early-returns while outstandingBlobsToFinish.length > 0, and
lastDurableCopyCursor — the value #683's gap path persists — is assigned only on that same
no-blob-in-flight path (:2425). The sender holds up to MAX_OUTSTANDING_BLOBS_BEING_SENT blobs in
flight continuously (:2903, replication_blobConcurrency, default 5), so during a blob-heavy copy
that array is essentially never empty — not at a commit, and not at a blob .finally.

Measured, fault-free control (1000 records, 50 KB file-backed blobs, flush cadence forced to
250 ms):

bulk copy starting from   22:06:29.745
bulk copy complete from   22:06:29.982     ← copy body: 237 ms
copy cursor advanced      22:06:30.031     ← finalization
copy cursor advanced      22:06:30.072     ← finalization

Zero persists during the copy body, with zero faults. Both advances are the finalization flush
(copyCompleteReceived forces flushNow once the stream ends and blobs drain).

(An earlier revision of this issue said the persist cadence "never runs." That is overstated — it
never runs during the copy; finalization persists do occur.)

Consequence: a copy that completes is fine — it finalizes and maybeFinishCopy removes the
cursor. A copy interrupted mid-flight finds the cursor still holding whatever preceded the copy,
so it restarts from scratch. This is fault-independent: peer restarts, component deploys, and bare
reconnects all trigger it, which is what #683 originally reported. It also explains the field
observation of cursors byte-identical to their values 24 h earlier — that is residue from an earlier
finalization, frozen ever since.

D2 — hasBlobGap is a one-way latch, so the copy never finishes

hasBlobGap (:2994) is set at :5531 and :5561 and never cleared. maybeFinishCopy() requires
!hasBlobGap (:2376), so a gapped receiver never exits copy mode even after copy complete from — it sits with the watchdog armed, re-copying the whole table every blobTimeout.

Measured (1000 records, sustained injected faults, 17 s quiet window with no writes on the
source): 3 watchdog fires → 4 full copies → 3000 blobs re-received, copyResumes=0 (it
never resumed from a cursor, because none was ever persisted).

D3 — the close-path persist is rejected by the ownership guard

retireInstance sets wsClosed = true (:5157) before abortInFlightBlobsOnClose (:5176). Those
aborts latch the connection-closed gap (:5531), so by the time the resulting .finally reaches
flushDurableCopyCursor, connectionSuperseded() (:2972) is already true and persistCopyCursor's
entry guard (:2443) rejects the write — on the very path the watchdog drives.

The fault supply is self-sustaining, and the watchdog feeds it

The minting step is settled by code, not inference:

• replication receives call saveBlob(localBlob) with no second argument, and
  saveBlob(blob, deleteOnFailure = false) (core/resources/blob.ts:912) therefore leaves
  deleteOnFailure false — an aborted replication receive cannot unlink, it must stub;
• abortInFlightBlobsOnClose (:711) destroys in-flight receives with a plain Error carrying
  replicationConnectionClosed but not sourceBlobUnavailable;
• createBlobReceiveStream(blobTimeout) gives idleTimeoutMs > 0.

Those are exactly the three conditions on core's stub branch (blob.ts:1085-1105). So every forced
reconnect with in-flight blobs stamps up to blobsInFlight.size PENDING stubs, each of which is a
503 to every peer copying from that node.

The serve→latch half is measured end to end (2-node test, 3 byte-identical PENDING stubs planted
on the source):

sourcePending=12   (exactly 4 retries × 3 stubs — BLOB_SEND_RETRY_DELAYS_MS exhausted, no heal)
blobSaveFailed=3   (reached the receiver)
unrecoverableClass=0   (classified transient → LATCHING branch, not advance-past)
cursorAdvanced=0

An abandoned stub cannot heal inside the ~3.75 s retry budget — and in fact never heals via
re-stream at all: every save mints a fresh fileId, so the reconnect's re-stream lands in a new file
and the stub is never overwritten (#481's "the re-stream overwrites this stub" assumption is false —
harper#2177's finding, disk-verified below). The record's dangling reference heals only when the
record itself is re-written, or re-delivered through copy-apply with harper#2177's repair.

Production attribution (30 min, docker logs, all four nodes)

node	latching (100% pending replication)	benign ENOENT	watchdog fires
yc0	0	0	0
cd5	10	98	4
e9v	100	808	4
v3t	478	935	4

Latching faults by source node:

• v3t ← 428 from e9v, 46 from yc0
• e9v ← 98 from cd5, 2 from yc0
• cd5 ← 8 from yc0, 2 from e9v

528 of 584 (~90%) come from sources that are themselves receivers — the minting mechanism. The
remaining supply (yc0, which never receives a base copy) is not transient mid-write contention as
this issue first assumed: every latching fault on this cluster traces to a fixed, persistent stub
population recurring at exactly the watchdog cadence — 23 distinct blobIds sourced from yc0
(92 faults/60 min, exactly 4 occurrences each = one per ~15 min), 217 distinct from e9v (849/60 min,
same signature). Two of yc0's were read directly off disk: 152-byte PENDING (0xfe) stubs, mtime
2026-08-13 23:44 — ~23 h old, minted before the 5.2.2 upgrade, still referenced by live records
(the fault lines name the record URLs). By fact (2) above even the smallest of these sets is enough
to keep a link wedged indefinitely.

Methodology note: the receiver logs whichever error rode the final forwarded frame, which need
not be the one that caused the failure. On this cluster they agree (100% of receiver-side latching
lines say pending replication), but in a synthetic stub test the source logged Blob pending replication while the receiver logged the other 503 arm (a write or replication receive is in flight). Both are BLOB_UNAVAILABLE_STATUS/503 and both latch identically, so the classification is
unaffected — but provenance is only reliable from source logs.

Reproduction

Two stress-gated tests (not yet in the CI matrix):

• integrationTests/cluster/copyCursorSustainedBlobFaults.test.mjs — fails on main. Sustained
  fault injection across a base copy; asserts the cursor persists at least once during the copy body,
  and that no full table is re-received in a quiet observation window.
• integrationTests/cluster/blobPendingStubLatchesReceiver.test.mjs — passes. Characterizes the
  supply chain: a source-held PENDING stub lands in the receiver's latching branch, never the
  advance-past branch. Deliberately does not assert cursorAdvanced === 0, which would make it a
  barrier against the fix.

Proposed fixes, with tradeoffs

A. Min-over-unresolved watermark — make the cursor advance during the copy (must-have)

Tag each blob at registration in receiveBlobs (:5483) with the copy cursor staged before it;
persist the cursorBefore of the lowest-numbered unresolved blob instead of early-returning.
Registration order equals copy-walk order (orderTablesForCopy :1496 → key order :4492 → blobs
stream inline with their record → ordered WS), so outstandingBlobsToFinish[0] is already the
minimum — no key comparator needed. Extend the same rule to gapped blobs and "one fault freezes
everything" becomes "one fault freezes everything after it."

This is the only fix that makes a multi-hour copy resumable, and D1 shows it is needed even at a zero
fault rate.

Tradeoffs / risks:

• Silent data loss if the watermark is ever one record too far ahead. The leader trusts afterKey
  absolutely — poisonedCopyCursorDataLoss.test.mjs characterizes a receiver that presents
  afterKey='row-0500', receives only rows 501+, and considers itself current. This is the same trade
  that produced Blob replication wedges permanently when source blob is gone (ENOENT) on an expiration cache table — held resume cursor never recovers #403 → Incomplete (truncated) source blob is misclassified as transient → permanently wedges replication cursor (ENOENT is handled; 'Blob is incomplete' is not) #429 → Sender returns BlobReadError 500/404 for in-flight blobs during catch-up — wedges receivers with no peer fallback #481. Two specific traps: pendingCopyCursor is nulled by
  persistCopyCursor (:2475, :2496), so it cannot be the snapshot source — a never-nulled staged
  cursor is required; and staging needs an explicit decode-order monotonicity guard so out-of-order
  commit completion can't make "first entry == min" pick a cursor ahead of an unresolved blob.
• Persist amplification. The cadence currently runs ~0 times during a copy; afterwards it runs
  every 5 s / 64 MB, each forcing a RocksDB flush() (memtable→SST). On a 376 GB copy that is
  thousands of forced flushes and added compaction pressure, on a path hasBlobGap never clears, so one blob-save fault condemns a link to base-copy on every restart #683 already measured degrading
  the serve path (6–10% of cache hits >1 s during copies). An identity-dedupe on the persisted cursor
  is load-bearing: without it, a gap-pinned watermark flushes on every commit for the whole timeout.
• Unbounded gapped-blob ledger. In-flight is capped at replication_blobConcurrency (5), but
  gapped is not — v3t takes 478 latching faults per 30 min. Needs a cap plus a conservative overflow
  floor.
• Constraint: do not add ledger fields to the persisted cursor object;
  unitTests/replication/discardMalformedCopyCursor.test.mjs pins its shape.

B. Let a held gap heal in-band (must-have)

Without it, maybeFinishCopy can never fire even with A, so the receiver never exits copy mode and
the watchdog keeps re-copying. Preferred: per-blob re-request over the existing connection via
getRecord({ …, blobRepairOnly: true }) (:6110, handler :3725, blob receive :3749) — the machinery
the blob repair sweep already uses (replication/replicator.ts:513), so no new protocol frame.

Tradeoffs / risks:

• Prerequisite — it can manufacture the fault it repairs. sendBlobs has three early-returns that
  send no error frame: dedupe (:5241), wsClosed (:5245), draining (:5249). The requester still
  creates a stream, registers it in-flight, and pushes the save into outstandingBlobsToFinish,
  bounded only by blobsTimer at blobTimeout — after which it fails, sets hasBlobGap, and core
  stamps a PENDING stub. Fix this before any higher-volume re-request path.
• Must rotate peers. Re-requesting from the peer that just 503'd re-reads the same stub — its
  GET_RECORD handler calls the same sendBlobs (:3694). Verified; without rotation it spins. Peer
  rotation is also what closes the root-source hole above: the other nodes likely hold healthy copies
  of the damaged blobs.
• Must never be awaited on the receive path, or it re-creates the rapid-reconnect stress: follower permanently loses replication backlog after restart churn (data loss; fast-skip ruled out) #426 ws.pause() ×
  in-flight-blob deadlock that :5039-5051 deliberately removed.
• Contends with the copy for the source's 5 send slots; needs a per-db concurrency clamp (1 while
  in copy mode) and backoff.
• Precedent to respect: replicator.ts:513-521 documents that a repair failure already latches
  hasBlobGap on the borrowed subscription connection, and repair_blob_data is broken on
  multi-worker (repair_blob_data always reports noConnection on multi-worker instances — it runs on the main thread, connections live on http workers #684).

C. Cut the fault supply at the source (high value, lower risk)

Before throwing PENDING, consult isBlobReceiveInFlight (core/resources/blob.ts:120) and wait for a
genuinely in-flight receive rather than 503-ing immediately; have abortInFlightBlobsOnClose enqueue
the stubs it is about to mint so they self-heal over the new connection. And for the persistent-stub
class above: when a copy read hits the source's own stub, no receive is in flight, and the stub is
older than some horizon, forward permanent (404/500) instead of 503 — the receiver then advances past
with #388 divergence debt instead of latching forever, which is the correct contract since the data
genuinely is unrecoverable from this source.

Tradeoffs / risks:

• Not sufficient alone for the transient class — by fact (2), a modest residue still wedges every
  link without A+B.
• Scope the wait to the replication send path. blob.ts:270 (bytes()) and :589 (stream())
  are the general read paths — the serve path uses them too. Making them wait up to
  getBlobReadTimeout (20 s) turns a fast failure into a long block on a path hasBlobGap never clears, so one blob-save fault condemns a link to base-copy on every restart #683 already measured
  as damaged during copies. The consumer's 500 ms cap and origin fallback likely absorb it, but
  changing the shared read is the riskier option.
• A waiting source holds a send slot. Several concurrent PENDING reads could stall 5/5 slots, and
  the copy-progress watchdog runs on the same blobTimeout — inventing a new wedge while fixing the
  old one. Cap the wait well below the retry budget, or don't hold the slot.

D. Fix the close-path persist (small; do alongside A)

Prefer the variant that does not reorder teardown: gate persistCopyCursor on socket identity
(connection.socket === ws) rather than wsClosed. Moving wsClosed = true after the aborts opens a
window where teardown is underway but sendBlobs (:5245), the drain loops, and
shouldRetrySourceBlobRead still think the socket is live. With the identity gate, a late write can
only move the cursor backward (redundant re-copy), never forward. #695 and #420 are both scars on
this exact path.

E. Observability (separate issue)

cluster_status exposes neither the cursor, nor inCopyMode, nor a held-gap indicator — the entire
investigation required a main-thread inspector. replication/knownNodes.ts:94-113 documents slots
13–15 as headroom, and readDbisCursorSync (:1159) already exists for the on-disk read.

Cross-cutting

• Re-point the watchdog. It arms on "a gap exists," which after A/B is no longer a wedge. Shipping
  A without re-pointing it at actual no-progress leaves it forcing a reconnect every blobTimeout —
  survivable now that the cursor is good, but still discarding in-flight work for nothing.
• Copy-mode exit timing changes. A gapped receiver can now leave copy mode, so monitorSync may
  mark a clone Available earlier than today. Desirable, but it is a behavior change on the
  clone-readiness path.
• Primary safety net for A: integrationTests/cluster/blobCopyInterruptionIntegrity.test.mjs
  (three-layer oracle across mid-copy SIGKILLs) and blobOrphanFullCopyConverges.test.mjs (guards
  that unrecoverable-at-source blobs still advance past).

Measured no-ops and things not to do

• replication_blobConcurrency: 1 — measured no-op. Run under sustained faults: cursorAdvanced=0, watchdogFires=3, re-received 3000 — identical to the default. Once hasBlobGap latches,
  flushDurableCopyCursor returns on the || hasBlobGap clause regardless of in-flight count.
• Serializing copies is not a fix. It would cut latching faults ~10x (528 of 584 come from
  receiver-sources), but by fact (2) one fault per cycle still zeroes it. Worth doing for noise, not
  as the answer.
• Do not lower REPLICATION_BLOBTIMEOUT — shared with blobsTimer and the chunk timeout; it would
  manufacture more faults. If the watchdog interval needs tuning it should get its own key. (PR Bank bulk-copy progress across blob-gap reconnects and repair missing blob files on copy re-delivery #701
  adds replication.blobGapReconnectMs for exactly this — note the deploy-ordering caveat on that PR:
  do not shorten it before harper#2177 is on the source nodes.)
• Do not advance past a PENDING blob unconditionally — that is Sender returns BlobReadError 500/404 for in-flight blobs during catch-up — wedges receivers with no peer fallback #481's silent-loss path. The
  age/registry-gated source-side reclassification in C is the narrow correct form.

Interim workaround while the pair lands: raising replication_blobTimeout above the copy
duration stops the watchdog killing the copy, and the stream then runs to completion — a run with 142
latching faults and no watchdog fires converged 1000/1000 with zero re-copies. Caveats: needs a
restart to apply; the timeout is shared with blobsTimer and the copy-progress watchdog; the receiver
still never exits copy mode, so this populates the corpus rather than resolving the link; and faulted
blobs are never re-requested, leaving #388 backfill debt. Additionally, operational today: the
persistent damaged sets are small and enumerable from the fault lines (23 records on yc0, 217 on e9v,
with URLs in the log lines) — re-rendering or deleting those records replaces the dangling references
with fresh blobs and removes the fault supply at its root, no code change required.

Also found while tracing (adjacent, not the cause)

• sendBlobs' three silent early-returns (:5241, :5245, :5249) — see B's prerequisite.
• replication/blobRepair.ts:35 passes the local table to getRecord, so request.table.tableId
  is a node-local id while the response is decoded against tableDecoders, which is keyed by the
  remote's ids (:3734, populated :3409). Latent decoder-slot clobber; ids usually coincide.

Caveats

The fix ranking is a reading of the code plus the measurements above; the persist-amplification cost
in particular is estimated, not measured. A, B, and C all remain within the single-point-cursor
representation — the durable state is really "frontier minus a set of holes," and each fix in the
#403/#429/#481/#683 lineage has been a better projection of that rather than the right shape. These
should resolve #699; they are probably not the last word.

[harper-joseph]

Reproduced deterministically on a local 2-node cluster

Test: integrationTests/cluster/copyCursorSustainedBlobFaults.test.mjs (stress-gated, not yet in the
CI matrix — it fails on main by design). Run against main @ v5.2.2 with core a2d633f2.

Setup. A holds 1000 pre-existing records in an authoritative 50 KB file-backed blob table (reuses
fixture-blob-orphan-full-copy-converges). B is fresh, carrying fixture-blob-fail-injector with
HARPER_TEST_BLOB_FAIL_INTERVAL=7 — every 7th receive-side blob save fails with ENOENT, sustained for
the whole run rather than in one bounded window. The injected error carries no sourceBlobUnavailable
marker, so it lands in receiveBlobs's else branch — hasBlobGap = true + watchdog armed, the
branch under test. B joins with add_node isLeader:true → base copy. replication_blobTimeout = 5s,
replication_copyCursorFlushIntervalMs = 250ms.

Result.

A=1000 B=1000 blobSaveFailed=571 cursorAdvanced=0 advancesAfterFirstFailure=0
watchdogFires=3 copyStarts=4 copyCompletes=4 copyResumes=0
blobChunks=1000->4000 (re-received 3000)

D1 confirmed, and stronger than this issue originally stated

cursorAdvanced=0 is the total, not the count after the first fault. Across a 1000-record
blob-heavy base copy — 1000 blobs received, flush cadence turned down to 250ms, so dozens of cadence
windows elapsed — persistCopyCursor logged success zero times, before any fault occurred. Same
result at 200 records in an earlier run.

Because the cadence had every opportunity to fire, this isolates the cause to the in-flight
early-return in flushDurableCopyCursor (outstandingBlobsToFinish.length > 0), and rules out the
flushNow cadence gate, copyFlushInFlight, and the persist backoff. That is the discriminating
measurement described in the Caveats section, answered locally — the field logpoint is no longer
needed to decide it.

The zero-progress consequence, in bytes

maybeFinishCopy() requires !hasBlobGap, and the gap is a one-way latch — so B never leaves copy
mode even after copy complete from. It sits there with the watchdog armed, and in a 17s observation
window with no writes at all on A:

• the watchdog fired 3 times
• B started 4 full copies (copyStarts=4, copyCompletes=4)
• copyResumes=0 — it never once resumed from a cursor, because none was ever persisted
• B re-received 3000 blobs, i.e. the entire table three more times

That is the same signature as the field cluster (8 forced reconnects, byte-identical cursor), just on
a 5s period instead of 900s. blobSaveFailed grew from 142 to 571 across those cycles — the re-copies
manufacture their own additional fault supply, consistent with the amplification described above.

Notes for anyone re-running this

• The receiver's copy-lifecycle log lines are copy starting from / copy complete from.
  Replicating all tables to and Resuming interrupted copy of database are emitted by the sender
  and read 0 on the receiver — an earlier revision of the test grepped for those and made a real copy
  look like no copy at all.
• Don't try to catch the copy mid-flight with a short blobTimeout: 1000 records / 50 MB completes in
  234 ms on loopback (fault window 21:23:03.661 → 21:23:03.895). Waiting out several watchdog
  periods after the copy "completes" is the reliable way to observe the wedge, and is a closer
  analogue of the field anyway.
• Convergence is not a usable oracle. B reaches 1000/1000 every time: a pinned cursor doesn't stop
  records streaming and committing, it only stops the resume point being written. The field copy
  (1.6M records / 376 GB) nets to zero because it never completes inside one connection and therefore
  always has to resume.

[kriszyp]

Root-caused — this turned out to be two defects, and the second is the "record exists but the blob file is missing" class from the top of the Slack thread. Fix pair is up:

• harper-pro#701 — Bank bulk-copy progress across blob-gap reconnects and repair missing blob files on copy re-delivery
• harper#2177 — Repair dangling blob references when a base copy re-delivers the record; add replication.blobGapReconnectMs

1. Banking was all-or-nothing (this issue's headline). Exactly as diagnosed here: flushDurableCopyCursor clamps on a per-connection boolean, and the #683 pre-gap snapshot only captures at an instant with zero blobs in flight — a blob-dense copy never reaches one, so nothing was ever staged to persist. The fix is your suggested direction #1: a per-walk-position watermark (durableCursor = highest staged position below min(first unsettled blob, first gapped blob)), mirroring lastDurableSequenceId semantics. A new integration test reproduces this cluster's livelock on clean 5.2.2 — 67 watchdog cycles, 408 injected faults, zero cursor movement, blobs pinned at 54/60 indefinitely — and converges in 2 cycles with the fix, resume keys strictly advancing.

2. The missing-blob-file bug (new finding). Building that reproduction surfaced permanent data loss the banking alone doesn't fix: a record whose blob save failed transiently keeps a reference to the PENDING stub, and when the resumed copy re-delivers the record with a successfully-saved fresh blob, core's copy-apply duplicate check skips the identical-version row — the dangling reference persists forever and the fresh blob is swept as an orphan. (#481's "the re-stream overwrites this stub" assumption never held: re-streams mint fresh fileIds.) harper#2177 repairs the reference on exactly that re-delivery. This should eliminate the missing-blob-file population at copy convergence, not just the copy churn.

Also in the pair, from your suggestion list: replication.blobGapReconnectMs decouples the #683 watchdog from blobTimeout (so you can shorten gap cycles on this cluster — e.g. 60000 — without making blob streams time out more aggressively; the shared-timeout concern in your "probably counterproductive" note was right and this removes it), and the "Resuming interrupted copy" log line now includes the resume key so banked progress is visible from logs (cluster_status exposure is a follow-up). Suggested direction #2 (in-band per-blob re-request) remains open as the follow-up for gap-heal latency; #3's fault supply (PENDING collisions) should mostly drain on its own once copies converge and stubs get repaired.