P2P path hardening plan for store-based P/D disaggregation

Context

The store-mediated P2P path (MetaServer discovery → RDMA READ fetch → local read cache) is intended to become the core of store-based prefill/decode disaggregation:

• Prefill (write side): save → write pipeline seals blocks (write_path.rs:288) → fire-and-forget registration to MetaServer → PegaEngine::flush_saves_and_registrations() (lib.rs:745) → signal KV-ready.
• Decode (read side): query_prefetch → local prefix scan → MetaServer query for the missing suffix → pick owner → QueryBlocksForTransfer (serving side pins blocks under a transfer lock) → RDMA READ into NUMA-local pinned slabs → rebuild SealedBlock into the read cache → re-register as a new owner → lease → H2D load.

#381 completed the skeleton: the flush barrier is the P/D handoff barrier, and P2pTransferService (p2p_service.rs) is the minimal serving surface for in-process embedders. The happy path works.

The gap: the path's semantics today are best-effort cache (any miss silently degrades to recompute). P/D disaggregation needs a high-hit-rate handoff. Every item below either closes a hit-rate hole or makes miss causes observable.

Scope note: transfer v2 is out of scope for this plan. The v1 READ engine (RcBackend) is the data plane throughout.

Phase 1 — P/D readiness (correctness & observability)

Small, independent changes; one PR each.

1.1 Owner failover

RdmaFetchStore::query_prefix picks the single owner with the longest prefix (rdma_fetch.rs:92-108) and never retries another owner. If the chosen owner is dead or has evicted the blocks, the whole fetch fails to recompute — even though other owners are listed. Dead nodes stay query-visible for up to node_stale_secs (30s, store.rs:433), so a crashed prefill node causes a hard TTFT cliff for that window.

Fix: on fetch failure, retry the next owner by descending prefix_len before falling back to recompute.

1.2 Query-side metrics

metrics.rs covers registration/removal/heartbeat/rdma_fetch, but there are no counters for MetaServer queries, hit rate, or miss attribution (no-owner vs stale-owner vs transfer-error). The first production question for P/D will be "why is the remote hit rate low" — currently unanswerable.

Fix: add query count, hit/miss counters, and per-cause fetch-failure counters.

1.3 failed_remote granularity

One short return (partial eviction on the serving peer) suppresses all RDMA for the entire request until GC (prefetch.rs:241-255, keyed by req_id). Under frequent P/D handoffs this amplifies transient jitter into full-request recompute.

Fix: key suppression by (peer, hash) instead of req_id.

1.4 flush() should report drops

The flush barrier guarantees "delivered or dropped after a failed attempt" (metaserver_client.rs:278-284). If the registration queue is full or the MetaServer is flapping, flush returns normally while whole batches were silently dropped — the decode side then misses 100% of that handoff.

Fix: return the dropped-registration count from flush() so the caller can decide (re-register, or signal degraded handoff) instead of ().

1.5 Make the slots/NUMA invariant structural in SealedBlock

The serving hot path builds wire slots with block.slots().iter().zip(block.slot_numas()) (p2p_service.rs:148). zip silently truncates to the shorter side — this already caused the 0-slot re-serve incident (fixed by e6407d1, but only on the construction side; the serialization-side landmine remains for any future constructor).

Fix: merge slots + slot_numas into a single Vec<(RawBlock, NumaNode)> field end to end, so mismatched lengths are unrepresentable.

Phase 2 — v1 data plane cleanups

Since v1 READ is the long-term data plane for this path, its hot-path debts are real work items:

1. find_local_mr linear scan (state.rs:148-163): O(n) scan over all registered regions per descriptor, while the remote side already uses sorted + binary search. Make local lookup symmetric.
2. Global state mutex held across the whole lookup/prepare phase of batch_transfer_async (mod.rs:456-513): serializes concurrent fetches from multiple peers. Move MR/rkey lookup out of the critical section.
3. Unused recv CQ / recv queue per session (session.rs:56,70-74): the path is one-sided only; delete.
4. .expect("session vec must exist") in the hot path (mod.rs:466): inconsistent state panics the caller instead of failing the fetch. Return an error and degrade to recompute.

Non-goals: busy-spin CQ polling and one-thread-per-QP stay as-is (deliberate latency/CPU tradeoff). Do not re-attempt parallelizing block rebuild — measured to regress due to Arc refcount contention; the READ-path floor is established.

Phase 3 — refactors

1. Unify SSD/RDMA prefetch block rebuild into one "staged pinned block builder". The two branches of run_prefetch_task (prefetch.rs:533-586) have near-parallel NUMA-slab allocation + segment bump-allocation implementations (rdma_fetch.rs:476-567 vs ssd_cache.rs:710) and have already diverged semantically (RDMA re-registers to MetaServer, SSD does not — prefetch.rs:262-268). This asymmetry is where the next 0-slot-class bug comes from.
2. Single source of truth for KV layout. Page-first vs layer-first is decided independently at register (worker.py:63-148), save (worker.py:704-723, _use_page_first at worker.py:812-834), and load (lib.rs:679-682), kept consistent only by matching boolean conditions. Introduce one layout type that all three sites derive from.
3. Rewrite pegaflow-transfer/README.md. It documents MooncakeTransferEngine, a UD/LZ4 control plane, and module files (sideway_backend.rs, control_protocol.rs, …) that no longer exist.

Open decisions

1. Handoff semantics: keep MetaServer-mediated discovery (miss → recompute) rather than adding a direct prefill→decode descriptor handoff. Recommendation: keep it — it avoids a new path, and Phase 1 items should push the miss rate low enough. Validate with the 1.2 metrics before revisiting.
2. nixl_connector retirement: it is a parallel P→D pull path (NIXL discovery, bypasses the store) sharing the same v1 engine. Once store-based P/D matures, decide whether it retires.
3. MetaServer restart backfill: nodes do not re-register resident keys after the MetaServer loses state (internode/README.md). For a cache this is slow warm-up; for P/D it is a cluster-wide handoff outage until new writes repopulate the catalog. Decide whether session-reset should trigger re-registration of resident keys.

Suggested order

Phase 1 items first (each is small and independently shippable; 1.1 + 1.2 + 1.4 together determine whether a store-based P/D demo has an explainable hit rate). Phase 2 items 1/2/4 follow. Phase 3 lands after P/D semantics are validated end to end.