Retain a blob file for the duration of its replication send

[kriszyp]

A blob resolved from the record being replicated is not opened until blob.stream(), and between resolving it and getting there the send can park on the outstanding-sends cap for an unbounded time. A write that supersedes the record in that gap reclaims the file, so the send reads a missing file and reports it to the peer as unrecoverable at source — the peer then advances its resume cursor past a record whose bytes it will never have (#403/#388), leaving a live record with a dangling blob reference until someone runs repair_blob_data.

This takes a core retention hold for the duration of the send and releases it on every exit path.

Pairs with HarperFast/harper#2145, which adds the hold API and the reference-aware reclamation queue underneath it; the core pointer is bumped to that commit. Reported context in HarperFast/harper#2134.

Scope

This closes the sender-side window (resolve → park → open) only. It does not cover a peer that is far enough behind that the blob was already reclaimed before the send began — that case has no file to hold, and is covered instead by the core retention window (storage.blobRetention), which an operator can raise to exceed their replication lag. Making retention properly lag-aware (retain until every peer's resume cursor has passed the superseding version) is a larger design and is not attempted here.

Verification

Route: existing blob-replication integration suites, run locally with the stress gate enabled.

• HARPER_RUN_STRESS_TESTS=1 npm run test:integration -- integrationTests/cluster/replicationBlobResyncOnFailure.test.mjs integrationTests/cluster/replicationBlobRepairAuthoritative.test.mjs integrationTests/cluster/blobSaveRejectionContainment.test.mjs
  • Receive-side blob save rejection containment — pass
  • Receive-side blob resume-cursor clamp on transient save failure — pass
  • Authoritative-table blob byte-integrity after receive-side save failure — fails, and fails identically on a clean origin/main tree with the core pointer restored (ECONNREFUSED 127.0.0.15:9933 during the restart-phase component deploy). Pre-existing in this local environment, not a regression from this change.
• The hold semantics themselves are unit-tested in the paired core PR (an explicit retention hold defers reclamation until it is released).
• No new test reproduces the park-window race deterministically — provoking it needs a send parked on the concurrency cap while a superseding write lands, which the current harness has no hook for.

For the human reviewer

Three entries, all inherited from the core PR's design rather than introduced here.

1. The hold cannot help a peer that is already too far behind. By the time the send starts, a blob reclaimed minutes ago has no file to hold, and holdBlobFile reports success anyway — so this changes nothing in exactly the lagging-peer case that produces the divergence in harper#2134. The lever for that case is storage.blobRetention, or lag-aware retention, which is not attempted. Worth ruling on whether shipping the narrow fix is worth it on its own.
2. The park window has no deterministic test. Provoking it needs a send parked on the outstanding-sends cap while a superseding write lands; the harness has no hook for that. The passing suites show no regression, not a proven fix.
3. The hold leaks if blob.stream() throws before the try/finally, bounded by core's 20-minute age cap. The pre-existing blobsBeingSent / outstandingBlobsBeingSent state leaks on that same path with no bound, so this is not a new class of problem — but it is a knowingly unfixed one.

_{Human-Review-Need: 4 (decisions: hold-layer-pro-vs-core, hold-through-park-vs-at-slot, null-hold-proceeds-vs-declines, no-submodule-bump) @ fc9b493}

[gemini-code-assist]

Code Review

This pull request updates the core subproject and introduces a mechanism to retain blob files during replication using holdBlobFile to prevent files from being reclaimed prematurely. The reviewer identified a potential resource leak where the blob hold and blobsBeingSent entry could leak if blob.stream() throws an error. Restructuring the function with an outer try...finally block was suggested to ensure robust cleanup and simplify the early exit paths.

[claude]

Reviewed; no blockers found.