Bound fluxnode memory growth: file-backed block index, header pruning, jemalloc (depends on #284)

[MorningLightMountain713]

⚠️ Merge dependency

PR #284 must merge first. This branch is based on feat/pon-vrf-integration and inherits its protocol bump (170022) and the VRF block-index fields (nodesVrfOutput is kept always-resident — the fork-choice comparator must never depend on prunable data, see "Fork choice" below). It will not apply to master until #284 lands.

Why fluxnodes need this

A Cumulus node has 8 GB of RAM, and that RAM is the product — it's what FluxOS sells to applications. Every megabyte fluxd pins is a megabyte no app can use, and unlike CPU (which idles between blocks), memory is occupied 24/7. On a stock build, fluxd was consuming ~1.9 GB of pinned anonymous memory — roughly ONE QUARTER of a Cumulus node's entire RAM — and the dominant component of that grows linearly with chain height, forever:

• mapBlockIndex holds one CBlockIndex per block (~2.67 M today, +~52 K/yr) in anonymous heap memory for the process lifetime.
• Anonymous memory can only be reclaimed to swap — useless on the many swapless or swap-starved nodes — so it is effectively pinned.
• Allocator tuning can't help: the memory is live. Without intervention, the footprint at 5 M blocks will be roughly double today's.

This branch makes fluxd's steady-state memory bounded: the chain keeps growing, our resident memory doesn't.

Measured (same height, 62 GB host, uncapped):

	stock node (salad)	this branch (squidward)
VmRSS	1906 MB	1075 MB
RssAnon (pinned, needs swap)	1855 MB	360 MB
RssFile (reclaimable, no swap)	52 MB	716 MB
Block index in…	anonymous heap	file-backed arena
Min footprint under no-swap pressure	~1.9 GB / OOM	~500 MB (measured)

Under a 500 MB MemoryHigh cgroup cap with MemorySwapMax=0, this branch ran fully synced with zero swap used; a stock node cannot fit in that envelope. On a Cumulus node this frees ~1.4 GB — about 18% of total RAM — back to apps, and the number stops growing with the chain.

The optimizations, and the reasoning behind them

All memory-optimization behavior is gated on fFluxnode; non-fluxnode builds behave like master.

1. The block index moves into a file-backed arena — why a file

The kernel can only evict a page to wherever its backing store is. Anonymous (heap) pages are backed by swap — no swap, no eviction, pinned forever. Pages backed by a file evict to that file under memory pressure and fault back in on access, no swap needed. So the fix for "the OS can't page this out" is to give the OS something to page it out to.

Concretely: CBlockIndex skeletons are placement-allocated from a segmented, file-backed arena — a single named scratch file (blockindex.arena in the datadir, mmap MAP_SHARED), each slot one CBlockIndex + its 32-byte hash. Hot entries (near the tip) stay in page cache and are exactly as fast as heap; cold entries (the millions of historic blocks, touched only by rescans/deep RPC) get evicted by the kernel under pressure and cost a page fault when touched. That converts the block index from pinned RssAnon into reclaimable RssFile — which is why the table above shows RssAnon dropping 1855 → 360 MB while RssFile rises.

Design points:

• The file grows ~128 MiB at a time (ftruncate + fixed mmap windows at increasing offsets; windows never move, so pointers stay valid). File size and VSZ track actual usage — no fixed reservation, no hard ceiling.
• Crash-safe by construction: the arena is pure scratch, rebuilt from leveldb every start (O_TRUNC at startup, unlink on clean shutdown).
• If a chunk can't be mapped (disk full, unsupported FS) → heap fallback, no abort — the node degrades to stock behavior instead of dying.
• At load the historic chain is proactively hinted cold (madvise MADV_COLD, MADV_DONTNEED fallback; a recent window stays hot), so its pages are reclaimed early rather than only under memory pressure.

2. CBlockIndex is split, and rebuildable data is pruned — why a split

Most of a CBlockIndex is data that exists byte-for-byte in the block files on disk (equihash solution, merkle root, nonce, collateral, block-sig). Keeping ~2.67 M copies of it resident is paying RAM for a second copy of the disk. The struct is split:

• Resident (small, NOT reconstructable from the block; read by init-time consensus checks): chain work/tx/skiplist/status, shielded value pools, nCachedBranchId, Sprout anchors, the VRF output, the cached PON hash.
• Prunable HeaderData (rebuildable from disk): merkle root, final sapling root, nonce, solution, nodes-collateral, block-sig.

HeaderData is resident only for a recent window of entries near the tip. At load it is freed for the whole historic chain (right after each entry's consistency check, not after ActivateBestChain — cutting the init-time memory transient ~2.1 GB → ~1.4 GB); at runtime each entry's HeaderData is freed once it ages past the window. It is rehydrated from the block files on demand (reorgs into pruned territory, header serving, rescans). Pruning frees the separately-allocated HeaderData (~112 bytes) per entry; the resident skeleton (~312 bytes) stays.

3. jemalloc — why the allocator matters here

The HeaderData prune frees ~2.5 million small allocations right after load. glibc malloc keeps those freed pages in its arena (they stay in RssAnon, invisible savings); jemalloc actually returns freed pages to the OS and fragments far less under this churn. Without it, optimization #2 would shrink our logical usage but not the number the node operator (and the OOM killer) sees.

Mechanics: fluxd links the system libjemalloc.so.2 dynamically (a malloc replacement needs dynamic linking for symbol interposition; the deb package will declare libjemalloc2 as a dependency). Detection is autotools (AC_CHECK_LIB), enabled by default, graceful fallback with a warning if absent, --disable-jemalloc to opt out; Windows/macOS keep their system allocators. Deploy check: ldconfig -p | grep jemalloc.

Fork choice stays resident (interaction with #284)

CBlockIndexWorkComparator tie-breaks same-height PON blocks via ComparePonForkChoice: VRF blocks compare by the committed nodesVrfOutput, legacy PON blocks by a PON hash cached on the index entry at load/creation (hashPON). Both scores are deliberately resident fields: recomputing from GetBlockHeader() would hash a zeroed collateral for pruned entries (wrong tie-break vs the network) and would mutate a comparator key in place for entries already inside setBlockIndexCandidates. No disk-format change.

Crash safety and correctness

The arena/prune machinery adds new failure surfaces a crash can expose — a flushed index entry may be header-pruned, and the fluxnode sync-marker and the coins DB are written on independent cadences, so a crash can leave them at different heights. These are the guarantees the branch holds:

• Crash recovery is exact. The fluxnode cache writes a sync-state marker at its persisted tip while the coins DB flushes on its own (~24 h) cadence, so a crash can leave the two at different heights. On restart the node rewinds fluxnode state along the marker's chain (DisconnectFluxnodeOnly) to the consistent height and replays to tip; it hard-fails rather than continue on a cache that already contains a block's effects, caps rewind depth at the undo-retention window, and aborts init before ActivateBestChain if recovery fails. An fFluxnodeCacheRecovered gate keeps init-time flushes (including RewindBlockIndex's forced flush) from overwriting the marker before recovery has read it.
• A pruned entry is never persisted as zeroes. The flush path rehydrates header data from disk before serialization and aborts on read failure; prunes skip header-only entries. PersistToDisk honours fForce, so a stale marker is repaired rather than silently kept.
• Reorgs into pruned territory recover rather than crash. Undo records are written for delegate-only blocks and retained by height across all chains, so fork-side records survive reorgs; pruned header data is rehydrated from the block files on demand.
• Pruning is invisible to the wallet. Confirmed transactions are not reported as conflicted after a fluxnode restart — the merkle check reads the rehydrated root, not a zeroed one.
• Header serving is cheap and lock-light. Serving reads only the header prefix from disk (no tx parse, no proof recheck) instead of a full block, and rehydration runs outside cs_main — a 2000-entry cmpheaders request no longer triggers 2000 full-block deserializations and Equihash rechecks under the lock.
• Consolidated pruned-header reads. The scattered pruned-header fallbacks now resolve through the exported GetFullBlockHeader, over a shared ReadBlockHeaderFromDisk primitive.

Compatibility / risk

• No consensus change. No reindex. On-disk CDiskBlockIndex serialization is byte-identical to Feat/pon vrf integration #284's (moved fields keep their serialized position/order; nodesVrfOutput's VRF-gated READWRITE preserved verbatim; memory-only fields stay memory-only).
• Non-fluxnodes: no optimization; functionally identical to Feat/pon vrf integration #284, with a small overhead from the HeaderData split (two allocations/block instead of one, ~50–85 MB at current height + a pointer indirection).
• Init still peaks ~1.4 GB (the arena bounds steady state, not init). A ≤2 GB node should init with headroom, then run capped.
• Reorg/invalidateblock into pruned territory triggers on-demand disk reads (rehydration) rather than crashes.
• Pruned + unreadable header fields surface at the header RPC/REST endpoints: when an entry is header-pruned and its block file cannot be re-read, getblockheader (non-verbose) errors rather than returning a stale in-memory header, blockheaderToJSON emits empty strings for the affected fields, and /rest/headers serializes the resident-only view. These differ from Feat/pon vrf integration #284 only on a missing/unreadable block file.
• New runtime dependency on Linux: system libjemalloc.so.2 (libjemalloc2 package) — graceful fallback if absent, but the memory numbers above assume it.

Testing

Done

• flux-gtest full suite (on the Feat/pon vrf integration #284 base): green, including Feat/pon vrf integration #284's 7 VRF tests (fork-choice, serialization, ECVRF prove/verify) which exercise the resident-score fork-choice merge.
• Custom gtests (all green): CDiskBlockIndex serialization round-trip (no-reindex guard), prune→restore serializes byte-identically, resident state incl. nCachedBranchId + cached PON hash survives the prune, CBlockIndexPool alloc/exhaustion/Contains/DestroyAll, forced PersistToDisk writes the sync marker on a clean cache, the resident VRF output (nodesVrfOutput) survives CBlockIndex copy/assignment and a v101 CDiskBlockIndex serialize round-trip (the disk-write path the fork-choice comparator depends on), and ExistsBlockUndoFluxnodeData distinguishes an absent fluxnode undo record from an empty one (the recovery missing-record check).
• New regtest qa/rpc-tests/fluxnode_cache_recovery.py (green): asserts all four recovery shapes — clean restart skips recovery; a stale marker is repaired exactly once; a marker behind the tip disconnects/replays to the same tip; a marker on a stale fork triggers the fluxnode-only rewind along the marker's chain and converges to the best tip.
• ThreadSanitizer dynamic race check (header serving): a regtest fluxnode built with -fsanitize=thread served /rest/headers from multiple HTTP worker threads over the header-pruned region while the validation thread pruned aged header data on every block connect (PruneAgedHeaderData → FreeHeaderData); TSan reported no data race on the prunable HeaderData, the block index, or the arena — confirming the serving path snapshots header fields under cs_main and rehydrates from disk lock-free, never touching pHeaderData off-lock.
• Live validation on mainnet fluxnodes: clean init/sync, value pools intact, ~500 MB under a 500 MB no-swap cgroup cap. A ~22 h soak of the pre-rebase (master-based) head across 6 nodes: RssAnon flat 356–467 MB, VmSwap 0, all at tip, host reboots recovered with "no recovery needed".
• Multi-day fleet soak on mainnet fluxnodes: RssAnon bounded in the ~425–500 MB band, VmSwap 0 (one memory-pressured host evicts cold anon pages to swap, by design), all at tip, no restarts.
• Mechanism re-confirmed live (current height): a direct read of the resident page state on a running mainnet fluxnode confirms the central claim in the field — the block index is resident as clean, file-backed page cache (blockindex.arena, Dirty=0, counted in RssFile, kernel-evictable), VmSwap 0; RssAnon oscillates within a bounded band and fully reclaims after each per-block allocation spike, with no monotonic growth across the soak.
• Wallet rescan over the bounded index (mainnet fluxnode on this head): a watch-only address was rescanned over a bounded window and its getreceivedbyaddress cross-checked to the satoshi against the independent address index (getaddressdeltas, coinbase-adjusted). This exercises the bounded-index historical read path — per-block CBlockIndex lookup + ReadBlockFromDisk + the GetFullBlockHeader nSolution disk fallback; RssAnon stayed bounded, VmSwap 0.
• Crash recovery under a real kill mid-rescan: the daemon was hard-killed (SIGKILL) twice with the wallet mid-write; both restarts recovered cleanly — chainstate behind the block files, fluxnode-cache UNDO PREPARE to a consistent height then replay to tip, no reindex and no corruption.
• Stress suite (2026-06-12, qa/stress/): restart storm (13/13 boots clean, "no recovery needed", post-settle RssAnon 318–449 MB); memory-pressure torture (500 MB MemoryHigh + MemorySwapMax=0, VmSwap 0 in every sample, arena pages evicting/faulting as designed); header-serving storm (exposed multi-second cs_main holds in getheaders serving → fixed in this branch: disk rehydration now runs outside cs_main, RPC stays at idle latency, 7–13 ms, while serving 40 k cold headers); rehydration churn and a 12-kill crash matrix on regtest; and mixed fork-choice at the VRF boundary — all green.
• Fresh-peer IBD served only by patched nodes: a brand-new node synced genesis→tip (progress=1.0) into a clean datadir sourcing only from two patched nodes, with no OOM and no crash and dbcache bounded (~89 MiB resident at tip); random-height block-hash cross-checks against a serving node matched throughout, and the serving nodes held at tip with bounded RssAnon.
• Also carries one test-only fix unrelated to the memory work: the CachedWitnessesCleanIndex case in test_wallet.cpp decremented note witnesses at a hardcoded height that could coincide with the chain tip under Flux's MAX_REORG_LENGTH=40; the decrement point is now derived from WITNESS_CACHE_SIZE so it stays below the tip. No production witness code is touched.

Remaining gate

The fleet soak, stress suite, fresh-peer IBD, and wallet-rescan validations above are complete. The only remaining gate is the #284 merge dependency (shared protocol bump 170022) — this PR must land after it.

Reviewer guidance — scrutinize

• CDiskBlockIndex::SerializationOp field order vs. Feat/pon vrf integration #284 (must be byte-identical, including the VRF gating).
• Every pHeaderData-> access: guarded, rehydrated, or pre-prune (see EnsureHeaderDataFromBlock/FromDisk, GetFullBlockHeader).
• The fFluxnode gating points (arena create, both prune sites).
• CBlockIndexPool lifetime: Contains()-based pool-vs-heap discrimination in cleanup paths; chunk-map failure → heap fallback; O_TRUNC/unlink cleanup; mmap-window offsets stay valid as the file grows.
• ComparePonForkChoice reads only resident fields (nodesVrfOutput, hashPON).

Deployment notes

• Linux runtime dependency: libjemalloc2 (ldconfig -p | grep jemalloc); the deb should declare it. Without it fluxd runs but RssAnon savings shrink (glibc retains freed pages).
• Pair with dbcache=200 (bounds the anonymous UTXO cache, the dominant remaining RssAnon; a per-tier tunable — lower for Cumulus/Nimbus).
• Exclude blockindex.arena from datadir backups/snapshots — a visible scratch file regenerated from leveldb each run.
• Rollback = restore the previous binary; no on-disk migration in either direction.

🤖 Generated with Claude Code