Skip to content

fix(testnet): recover safely from source reorgs - #151

Open
sprites0 wants to merge 10 commits into
node-builderfrom
fix/pseudo-peer-recover-evicted-blocks
Open

sprites0 wants to merge 10 commits into
node-builderfrom
fix/pseudo-peer-recover-evicted-blocks

Conversation

@sprites0

@sprites0 sprites0 commented Sep 1, 2026

Copy link
Copy Markdown
Collaborator

Summary

Enable Hyperliquid testnet nodes to recover from source reorgs within a 256-block window while preserving immediate finality on mainnet.

Changes

  • keep a 256-block reorg window on testnet (998)
  • continue finalizing every selected head on mainnet (999)
  • refresh stale cached blocks and maintain consistent hash indexes
  • reject stale queued responses during source reorgs
  • preserve finalization state during the reorg-depth transition
  • read sync blocks and receipts from one consistent database snapshot
  • resolve prefetched parent hashes and recover evicted blocks by hash

Safety

Reorg support is selected from the chain ID. Mainnet and unknown chains use immediate finality; only testnet receives the 256-block reorg window.

Testing

  • consensus finalization-policy tests
  • block-import service tests
  • source reorg tests
  • appended reorg coverage

@sprites0
sprites0 force-pushed the fix/pseudo-peer-recover-evicted-blocks branch 4 times, most recently from 90b1f85 to 18be869 Compare September 1, 2026 05:35
@sprites0 sprites0 changed the title fix(pseudo-peer): recover evicted blocks on hash lookup fix(testnet): recover safely from source reorgs Sep 1, 2026
@sprites0
sprites0 force-pushed the fix/pseudo-peer-recover-evicted-blocks branch 3 times, most recently from 8303058 to b43b8ef Compare September 1, 2026 05:52
Keep enough canonical history available for deep reorgs, avoid prematurely finalizing imported heads, and serve blocks with receipts from one consistent database snapshot.
Reth can begin a falling header request at a child's parent, but the reorg cache refactor stopped recording that parent mapping and caused repeated empty responses.

Index each fetched block's parent at the preceding height and cover the lookup with a focused regression test.
@sprites0
sprites0 force-pushed the fix/pseudo-peer-recover-evicted-blocks branch from b43b8ef to 4e02d2b Compare September 3, 2026 15:21
@sprites0
sprites0 marked this pull request as ready for review September 3, 2026 15:27
@sprites0
sprites0 force-pushed the fix/pseudo-peer-recover-evicted-blocks branch 3 times, most recently from 18b3471 to b8a5ca4 Compare September 4, 2026 00:26
sprites0 and others added 7 commits September 4, 2026 03:55
get_by_hash only served from the `blocks` LRU, which holds
BLOCK_CACHE_LIMIT (100k) entries. The hash index holds ten times that
(HASH_INDEX_LIMIT, 1M) and falls back to the node's HeaderNumbers table
on a miss, so a hash stays resolvable long after its body has been
evicted. Every peer request for such a hash failed with "Block not
cached for hash", even though the block was still reachable by number.

Resolve the hash to a height and refetch instead: serve from
gap_blocks when the height is one of the hardcoded testnet gaps,
otherwise call the source under a 30s timeout so a stalled fetch cannot
wedge the request loop.

Recovery is only allowed to install a block that is provably the one
asked for. The source is a by-number API, so it can legitimately return
a different block than requested after a reorg; three checks reject
that. The returned block must be at the requested height and carry the
requested hash, and — under the canonical_hashes write lock — must not
contradict a hash already recorded for that height. The last check is
what closes the race: an in-flight recovery that finishes after
cache_block has refreshed the height loses, rather than overwriting the
newer canonical block with the one it happened to fetch.

The staleness check in BlockImport::poll was using get_by_hash's error
as its signal, which is wrong in both directions now. It would trigger
a source fetch on what is only a liveness test, and a hash that reorged
away but still resolves through the DB index would come back Ok and be
treated as live. Add is_cached_canonical_hash, which answers exactly
that question with no I/O: the body is cached and its hash is still
canonical at its height.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011Z9cmqnwb9tL9kxkGM5Ccb
@sprites0
sprites0 force-pushed the fix/pseudo-peer-recover-evicted-blocks branch from 5b1676c to 6a8b71c Compare September 4, 2026 07:55
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant