Constrain replication mesh when the system database is replicated

[kriszyp]

Summary

Lets the system database replicate (node discovery + users/roles/schema propagation) without every aggregation node opening direct connections to every discovered peer. A node with directional config routes now advertises a directional hdb_nodes self-record (derived from those routes via computeSelfReplicates, opt-in — a node with no directional routes keeps legacy replicates: true) instead of a blanket replicates: true. The existing controlled-flow gates (harper-pro#498 — shouldReplicateFromNode receive-side, the send-authority gate) then consult the discovered peer's advertised directions instead of a permissive boolean.

Purpose

Hierarchical topology: core(1) ← middle(3) ← roadside(30), tight transitive routing (data aggregates roadside→middle→core, no cross-cluster connections) and system replicated so node discovery + central config propagate without hand-configuring every node. Previously mutually exclusive — replicating system caused every discovered node to open a direct socket to every other (core would hold ~30 direct roadside sockets). Docs (companion PR below) currently say to disable system to get tight topology; this removes that tradeoff.

What changed

• replication/subscriptionManager.ts — computeSelfReplicates(routeList) + getConfiguredRoutes() extracted/exported; ensureThisNode compares replicates structurally (not just url/shard) so a stale/legacy self-record is corrected on the next boot after a route change.
• replication/setNode.ts — selfReplicatesForNodeWrite() preserves an existing directional self-record instead of forcing true on add_node/set_node; both handlers drop the unconditional sends: true stamped on a directional PEER record (it was short-circuiting the receive gate's allow-list to "feeds every database").
• replication/knownNodes.ts — mergeReconstructedNode preserves a peer's last-known directional replicates through a transient hdb_nodes decode miss, applied at all three reconstruct/merge call sites (including the copyApply/base-copy reload scan path, harper-pro#489).
• Tests: unit coverage for computeSelfReplicates/mergeReconstructedNode; integration tests for transitive 3-tier routing, per-database opposite directions, and a churn/hardening test proving no reconnect/socket churn to an excluded discovered peer under sustained system+data mutation.
• replication/DESIGN.md documents the mechanism and its boundaries.

Where to look / open items

• replication/replicationConnection.ts leading-duplicate-skip qualifies checks (~4318, ~4578) are not updated for the dropped sends: true: a directional peer ({sendsTo:[...]}, no .sends) now evaluates qualifies=false there, so it's omitted from the multi-hop dedup exclusion list. Effect is a subscriber may receive some records via more than one path — idempotent, not data loss (replication applies by sequence). Deliberately left as-is rather than adding || sendsTo?.length, which would be the unsafe direction (over-exclusion risks dropping records — cf. the perf(replication): fast-skip leading duplicates on resume #370/Replication: transitive/proxied re-delivery floods peers with already-applied out-of-order writes (reduce volume; complements harper#1310) #399 leading-dup-skip family). A correct fix needs a subscriber+database-aware check and is a separate change. Documented in DESIGN.md.
• getConfiguredRoutes() reads a main-thread-only module global. setNode/addNodeBack can run on any worker; on a brand-new node whose main-thread self-record hasn't been written yet, a worker-handled add_node falls back to legacy true for this node's own row. Self-heals via ensureThisNode's structural-diff rewrite on next boot; never hit by the primary config-route topology (self-record is written from the main thread at boot). Documented in DESIGN.md.
• Central visibility of every node is not guaranteed — an aggregation node may not list every distant leaf in its own hdb_nodes registry (the registry relay differs from data relay). Does not cause a connection either way; documented as an accepted tradeoff.
• Cross-model review (Codex + Sonnet 5; Gemini agy was unavailable) surfaced one real bug (a third decode-recovery merge site I'd missed, now fixed) and the two open items above — no other material findings.

Docs

Companion PR: HarperFast/documentation#583

Testing

Unit tests (computeSelfReplicates/mergeReconstructedNode) 14/14 pass on a real tsc build. The churn/hardening integration test and the per-database repro pass on the real build. The transitive 3-tier repro is syntactically valid and converted from print-only to assertions, but could not get a clean local run due to contention from other test processes on the shared dev box — needs CI to confirm.

Generated by Claude (Opus 4.8), cross-reviewed by Codex and Claude Sonnet 5.

[gemini-code-assist]

Code Review

This pull request introduces a directional self-record mechanism to allow the replication of the system database without collapsing hierarchical topologies into a full mesh. It derives directional routing rules from configuration routes, preserving constrained connections while allowing transitive schema and role propagation. The feedback recommends defensive programming enhancements: guarding against a falsy node name during early boot in setNode.ts to prevent potential runtime crashes, and using Array.isArray checks in subscriptionManager.ts to robustly handle potentially misconfigured non-array routing entries.

[cb1kenobi]

If I'm understanding this function correctly, this isn't self replicating, it's computing the "replicates" object for an instance, in which case, this is a clever, but poor name.