Skip to content

bug: already-subscribed core node never receives a freshly published KA (VM divergence) and its random-sampling loop spams kc-not-synced/KCNotFoundError every 5s #1098

Description

@Bojan131

Bug Description

A core node that was already subscribed and fully caught up on a CG never received a KA published seconds later by another node — its VM view stayed empty for that KA indefinitely (checked at +10s, +40s, +several minutes), while two other nodes had it within seconds. Compounding it, every core node that misses a KA gets stuck in a random-sampling warn loop: [rs.tick.kc-not-synced] ... KCNotFoundError logged every ~5 seconds, indefinitely (39-64 occurrences per node within ~25 minutes), because the RS loop keeps drawing challenges for a KA the node never stored.

Expected Behavior

Actual Behavior

  • Edge node5 published edge-note to CG qa-main-test (status confirmed, txHash + UAL returned). VM query SELECT ... FILTER(CONTAINS(STR(?o),"Edge")) with view: "verifiable-memory": node1 → 1 row, node3 → 1 row, node2 → 0 rows, permanently (node2 had subscribed + completed catch-up minutes earlier with peersSucceeded 5/5).
  • node2 log fills with [rs.tick.kc-not-synced] {"kaId":"5369...763","cgId":"3","periodStart":"1600","err":"KCNotFoundError"} [WARN] every ~5s.

Relevant Logs / Error Output

2026-06-10 11:10:21 connect ee062727 [DKGAgent] [rs.tick.kc-not-synced] {"kaId":"53692664000115763942479723237650470929155136551880323364030089406489307381763","cgId":"3","periodStart":"1600","err":"KCNotFoundError"} [WARN]
(repeats every ~5s; 62 occurrences on node2, 39/42/64 on nodes 1/3/4 for various KAs)
[ProtocolRouter] stream closed by peer before response on /dkg/10.0.2/sync from oKowkGuR   (31-42 occurrences per core node)

Steps to Reproduce

  1. Clean devnet 6 nodes; on node1 create + register CG; subscribe node2 (POST /api/context-graph/subscribe) and wait for catch-up status: done.
  2. Publish a KA into that CG from another node (edge node5 used here): create with quads → swm/sharevm/publish (confirmed).
  3. Query view: "verifiable-memory" on node1/2/3 — node2 (in our runs) never gets the rows.
  4. Tail node2's daemon.log → kc-not-synced warn loop.

Possible Root Causes

Likely connected to the stale core-peer classification reported in #1093 — node2's knownCorePeerIds contained only 1 peer, and catch-up/sync ordering is core-preferred, so node2's sync paths may be repeatedly hitting the wrong/limited peers (its log also shows recurring stream closed by peer before response on /dkg/10.0.2/sync). Chain-driven VM reconciliation should have backstopped this regardless (node2 sees the KCCreated/KnowledgeAssetRegisteredToContextGraph chain events in its log and even logs "Phase B: KACG nudge ... -> reconcile" for them) — so the reconcile path appears to nudge but not actually materialize the KA.

DKG daemon / version

10.0.0-rc.17, commit 809cf3a (origin/main 2026-06-10)

Surface / Component

dkg-agent sync / VM reconciliation / random-sampling loop

Notes / Related Issues

Related: #1093 (ACK pool / stale core classification, same QA pass), #886 (late subscribers miss historical VM — different case: this node was subscribed before the publish), #1064 (stale protocol advertisement gating).

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't workingfix-in-flightFix exists on an open PR — close after mergepre-mainnetMust land before mainnet launchpriority:highMust-fix: protocol correctness, security, economics, or headline flow broken

    Type

    No type

    Fields

    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions