Symptom
A node that subscribes to a public context graph after another node has already published a KA on chain will never see that KA in its Verified Memory. The KA stays on chain forever, but the local _verified_memory/* graph is never populated, and there is no recovery path (no API, no background reconciler) that closes the gap.
This is reproducible against release/rc.12 (fa0ea47d) and is not specific to one CG.
Concrete repro from the wild
CG: 0xE5B88968Ed464F4e3f5354C54DFAB9e39dfEAfBd/monday-fun-facts (public, onChainId=3)
Curator's publish tx (Base Sepolia): 0xbf49ea9b97…58654fc
Replica node (mine, 12D3KooWCcZHkkC5dp6awNZrRgcMmRdsVn8ndUycDUKBtTe2QJqo):
- First became aware of the CG at 12:24:56Z (auto-subscribed via discovery gossip)
- Successfully ran SWM sync against many peers (most recent: `SWM sync … 133 data + 690 meta triples`)
- Daemon log contains zero lines matching
KCCreated, KnowledgeAsset, verify, or _verified_memory for this CG
- Local store partition listing:
| partition |
triples |
…/_meta |
57 (only my own assertion lifecycle) |
…/_shared_memory |
133 |
…/_shared_memory_meta |
779 |
…/assertion/0x9835…/mpv5g01t-0-sync-flow.md |
8 (my own extraction provenance) |
…/_verified_memory/* |
0 graphs, 0 triples |
- `/api/query?view=verified-memory` returns
count=0
The chain tx is real and confirmed, the CG is public, the SWM sync works, the curator's peer is reachable — and yet VM stays empty. There's nothing the operator can do via API or UI to recover.
Root cause — two missing mechanisms
There are exactly two code paths that could populate _verified_memory/* for a foreign publisher's KA, and neither does:
1. ChainEventPoller.handleBatchCreated (`packages/publisher/src/chain-event-poller.ts`)
Sees `KCCreated` events and only calls `publishHandler.confirmByMerkleRoot(merkleRoot, …)`. That function exists to promote a local tentative publish to confirmed; it returns `false` (no-op) when the merkle root doesn't match any pending publish on this node. So for every `KCCreated` emitted by another publisher, the poller logs the block + range and moves on without fetching the data.
There is no "is this CG one I'm subscribed to → trigger peer fetch" branch.
2. CatchupRunner (`packages/cli/src/catchup-runner.ts`)
`CatchupJobResult` exposes exactly two phases: `durable` and `sharedMemory`. There is no `verifiedMemorySynced` counter, no `diagnostics.verifiedMemory.*`, and the underlying sync protocol handler does not enumerate _verified_memory/<vmId> graphs when serving a peer. So even when a late subscriber initiates catchup against a peer that DOES hold the VM locally, the VM is never offered or asked for.
Why gossip doesn't save you either
V10's live path is the publisher's `GossipPublishHandler` finalization broadcast. That works fine while a peer is online and subscribed at publish-time, but gossip aging means anyone who subscribes more than a few minutes after the publish misses it permanently. In this repro, the subscribe was 48 minutes after publish — well outside any gossip retention window — and the daemon log confirms no finalization gossip with on-chain proof was ever received.
Why /api/verify doesn't rescue you
`agent.verify()` (`packages/agent/src/dkg-agent.ts`, around line 15784) first reads `?kc dkg:merkleRoot ?root ; dkg:batchId ` out of the LOCAL `/_meta` graph. The meta records would normally be seeded by either your own publish path or an inbound gossip — neither of which happens for a late subscriber — so `/api/verify` reliably 404s with "Batch N not found in context graph …".
Proposed fix
Two clean approaches, pick one (or both). My recommendation is (B) — it composes with the existing peer-sync pattern instead of widening the chain poller's responsibility surface, and it works the same for nodes that join after a long offline gap, not just first-time subscribers.
(A) Pull-on-chain-event in ChainEventPoller
In `handleBatchCreated`, after `confirmByMerkleRoot` returns `false`, check whether the event's CG (derivable from the contract address + `KCCreated` payload) is a CG this node is subscribed to. If yes, enqueue a peer fetch for the KA's data + meta keyed on `merkleRoot`, validate against the on-chain merkle root before insert, and write `/_meta` + `/_verified_memory/`.
- Pro: chain is single source of truth — convergence guaranteed
- Con: widens `ChainEventPoller`'s responsibility (currently it's purely a confirmation poller); needs peer-fetch wiring it doesn't have today
(B) Add a verifiedMemory phase to CatchupRunner ← recommended
Mirror the existing durable and sharedMemory phases:
- Sync responder enumerates
<cg>/_verified_memory/<vmId> graphs for the requested CG (gated on the CG being public OR the requesting peer being allowlisted, same predicate the durable/swm phases already use)
- Catchup client requests them, validates each graph's merkle proof against the on-chain merkle root (using the same chain adapter the publisher already uses), and writes on success
- New `diagnostics.verifiedMemory.{fetched,inserted}{Meta,Data}Triples` counters + `verifiedMemorySynced` top-level
This composes with subscribe-time catchup, periodic re-catchup, and the existing `subscribed: true / synced: true` lifecycle. It also benefits offline-gap recovery, not just first-time subscribers.
Acceptance criteria
Why this matters now (rc.12)
This is the only memory layer that silently and permanently fails to converge for a legitimate user action (subscribe to a public CG). Both WM and SWM have working recovery paths via catchup; VM does not. Operators have no API surface to recover, so the only workaround is "ask the curator's node to be online when you happen to subscribe" — which doesn't match the V10 mental model of chain-anchored data being durable and recoverable.
Symptom
A node that subscribes to a public context graph after another node has already published a KA on chain will never see that KA in its Verified Memory. The KA stays on chain forever, but the local
_verified_memory/*graph is never populated, and there is no recovery path (no API, no background reconciler) that closes the gap.This is reproducible against
release/rc.12(fa0ea47d) and is not specific to one CG.Concrete repro from the wild
CG:
0xE5B88968Ed464F4e3f5354C54DFAB9e39dfEAfBd/monday-fun-facts(public,onChainId=3)Curator's publish tx (Base Sepolia):
0xbf49ea9b97…58654fc0xE5B88968…dfEAfBd0x197e89973184e151592cf1273ca5b8dcb817dc51a9c9d2b5b922d8b94b028590mpuybndk-40Replica node (mine,
12D3KooWCcZHkkC5dp6awNZrRgcMmRdsVn8ndUycDUKBtTe2QJqo):KCCreated,KnowledgeAsset,verify, or_verified_memoryfor this CG…/_meta…/_shared_memory…/_shared_memory_meta…/assertion/0x9835…/mpv5g01t-0-sync-flow.md…/_verified_memory/*count=0The chain tx is real and confirmed, the CG is public, the SWM sync works, the curator's peer is reachable — and yet VM stays empty. There's nothing the operator can do via API or UI to recover.
Root cause — two missing mechanisms
There are exactly two code paths that could populate
_verified_memory/*for a foreign publisher's KA, and neither does:1.
ChainEventPoller.handleBatchCreated(`packages/publisher/src/chain-event-poller.ts`)Sees `KCCreated` events and only calls `publishHandler.confirmByMerkleRoot(merkleRoot, …)`. That function exists to promote a local tentative publish to confirmed; it returns `false` (no-op) when the merkle root doesn't match any pending publish on this node. So for every `KCCreated` emitted by another publisher, the poller logs the block + range and moves on without fetching the data.
There is no "is this CG one I'm subscribed to → trigger peer fetch" branch.
2.
CatchupRunner(`packages/cli/src/catchup-runner.ts`)`CatchupJobResult` exposes exactly two phases: `durable` and `sharedMemory`. There is no `verifiedMemorySynced` counter, no `diagnostics.verifiedMemory.*`, and the underlying sync protocol handler does not enumerate
_verified_memory/<vmId>graphs when serving a peer. So even when a late subscriber initiates catchup against a peer that DOES hold the VM locally, the VM is never offered or asked for.Why gossip doesn't save you either
V10's live path is the publisher's `GossipPublishHandler` finalization broadcast. That works fine while a peer is online and subscribed at publish-time, but gossip aging means anyone who subscribes more than a few minutes after the publish misses it permanently. In this repro, the subscribe was 48 minutes after publish — well outside any gossip retention window — and the daemon log confirms no finalization gossip with on-chain proof was ever received.
Why
/api/verifydoesn't rescue you`agent.verify()` (`packages/agent/src/dkg-agent.ts`, around line 15784) first reads `?kc dkg:merkleRoot ?root ; dkg:batchId ` out of the LOCAL `/_meta` graph. The meta records would normally be seeded by either your own publish path or an inbound gossip — neither of which happens for a late subscriber — so `/api/verify` reliably 404s with "Batch N not found in context graph …".
Proposed fix
Two clean approaches, pick one (or both). My recommendation is (B) — it composes with the existing peer-sync pattern instead of widening the chain poller's responsibility surface, and it works the same for nodes that join after a long offline gap, not just first-time subscribers.
(A) Pull-on-chain-event in
ChainEventPollerIn `handleBatchCreated`, after `confirmByMerkleRoot` returns `false`, check whether the event's CG (derivable from the contract address + `KCCreated` payload) is a CG this node is subscribed to. If yes, enqueue a peer fetch for the KA's data + meta keyed on `merkleRoot`, validate against the on-chain merkle root before insert, and write `/_meta` + `/_verified_memory/`.
(B) Add a
verifiedMemoryphase toCatchupRunner← recommendedMirror the existing
durableandsharedMemoryphases:<cg>/_verified_memory/<vmId>graphs for the requested CG (gated on the CG being public OR the requesting peer being allowlisted, same predicate the durable/swm phases already use)This composes with subscribe-time catchup, periodic re-catchup, and the existing `subscribed: true / synced: true` lifecycle. It also benefits offline-gap recovery, not just first-time subscribers.
Acceptance criteria
/api/query?view=verified-memoryreturns the union of all N KAs' triples once catchup completesdurable/sharedMemorysync diagnosticsmonday-fun-facts) yields a non-zero VM triple count on a fresh subscriber within one catchup cycleWhy this matters now (rc.12)
This is the only memory layer that silently and permanently fails to converge for a legitimate user action (subscribe to a public CG). Both WM and SWM have working recovery paths via catchup; VM does not. Operators have no API surface to recover, so the only workaround is "ask the curator's node to be online when you happen to subscribe" — which doesn't match the V10 mental model of chain-anchored data being durable and recoverable.