Bug Description
A core node that was already subscribed and fully caught up on a CG never received a KA published seconds later by another node — its VM view stayed empty for that KA indefinitely (checked at +10s, +40s, +several minutes), while two other nodes had it within seconds. Compounding it, every core node that misses a KA gets stuck in a random-sampling warn loop: [rs.tick.kc-not-synced] ... KCNotFoundError logged every ~5 seconds, indefinitely (39-64 occurrences per node within ~25 minutes), because the RS loop keeps drawing challenges for a KA the node never stored.
Expected Behavior
Actual Behavior
- Edge node5 published
edge-note to CG qa-main-test (status confirmed, txHash + UAL returned). VM query SELECT ... FILTER(CONTAINS(STR(?o),"Edge")) with view: "verifiable-memory": node1 → 1 row, node3 → 1 row, node2 → 0 rows, permanently (node2 had subscribed + completed catch-up minutes earlier with peersSucceeded 5/5).
- node2 log fills with
[rs.tick.kc-not-synced] {"kaId":"5369...763","cgId":"3","periodStart":"1600","err":"KCNotFoundError"} [WARN] every ~5s.
Relevant Logs / Error Output
2026-06-10 11:10:21 connect ee062727 [DKGAgent] [rs.tick.kc-not-synced] {"kaId":"53692664000115763942479723237650470929155136551880323364030089406489307381763","cgId":"3","periodStart":"1600","err":"KCNotFoundError"} [WARN]
(repeats every ~5s; 62 occurrences on node2, 39/42/64 on nodes 1/3/4 for various KAs)
[ProtocolRouter] stream closed by peer before response on /dkg/10.0.2/sync from oKowkGuR (31-42 occurrences per core node)
Steps to Reproduce
- Clean devnet 6 nodes; on node1 create + register CG; subscribe node2 (
POST /api/context-graph/subscribe) and wait for catch-up status: done.
- Publish a KA into that CG from another node (edge node5 used here): create with quads →
swm/share → vm/publish (confirmed).
- Query
view: "verifiable-memory" on node1/2/3 — node2 (in our runs) never gets the rows.
- Tail node2's daemon.log → kc-not-synced warn loop.
Possible Root Causes
Likely connected to the stale core-peer classification reported in #1093 — node2's knownCorePeerIds contained only 1 peer, and catch-up/sync ordering is core-preferred, so node2's sync paths may be repeatedly hitting the wrong/limited peers (its log also shows recurring stream closed by peer before response on /dkg/10.0.2/sync). Chain-driven VM reconciliation should have backstopped this regardless (node2 sees the KCCreated/KnowledgeAssetRegisteredToContextGraph chain events in its log and even logs "Phase B: KACG nudge ... -> reconcile" for them) — so the reconcile path appears to nudge but not actually materialize the KA.
DKG daemon / version
10.0.0-rc.17, commit 809cf3a (origin/main 2026-06-10)
Surface / Component
dkg-agent sync / VM reconciliation / random-sampling loop
Notes / Related Issues
Related: #1093 (ACK pool / stale core classification, same QA pass), #886 (late subscribers miss historical VM — different case: this node was subscribed before the publish), #1064 (stale protocol advertisement gating).
Bug Description
A core node that was already subscribed and fully caught up on a CG never received a KA published seconds later by another node — its VM view stayed empty for that KA indefinitely (checked at +10s, +40s, +several minutes), while two other nodes had it within seconds. Compounding it, every core node that misses a KA gets stuck in a random-sampling warn loop:
[rs.tick.kc-not-synced] ... KCNotFoundErrorlogged every ~5 seconds, indefinitely (39-64 occurrences per node within ~25 minutes), because the RS loop keeps drawing challenges for a KA the node never stored.Expected Behavior
Actual Behavior
edge-noteto CGqa-main-test(status confirmed, txHash + UAL returned). VM querySELECT ... FILTER(CONTAINS(STR(?o),"Edge"))withview: "verifiable-memory": node1 → 1 row, node3 → 1 row, node2 → 0 rows, permanently (node2 had subscribed + completed catch-up minutes earlier withpeersSucceeded 5/5).[rs.tick.kc-not-synced] {"kaId":"5369...763","cgId":"3","periodStart":"1600","err":"KCNotFoundError"} [WARN]every ~5s.Relevant Logs / Error Output
Steps to Reproduce
POST /api/context-graph/subscribe) and wait for catch-upstatus: done.swm/share→vm/publish(confirmed).view: "verifiable-memory"on node1/2/3 — node2 (in our runs) never gets the rows.Possible Root Causes
Likely connected to the stale core-peer classification reported in #1093 — node2's
knownCorePeerIdscontained only 1 peer, and catch-up/sync ordering is core-preferred, so node2's sync paths may be repeatedly hitting the wrong/limited peers (its log also shows recurringstream closed by peer before response on /dkg/10.0.2/sync). Chain-driven VM reconciliation should have backstopped this regardless (node2 sees theKCCreated/KnowledgeAssetRegisteredToContextGraphchain events in its log and even logs "Phase B: KACG nudge ... -> reconcile" for them) — so the reconcile path appears to nudge but not actually materialize the KA.DKG daemon / version
10.0.0-rc.17, commit 809cf3a (origin/main 2026-06-10)
Surface / Component
dkg-agent sync / VM reconciliation / random-sampling loop
Notes / Related Issues
Related: #1093 (ACK pool / stale core classification, same QA pass), #886 (late subscribers miss historical VM — different case: this node was subscribed before the publish), #1064 (stale protocol advertisement gating).