Skip to content

feat(ocap-kernel): look an endpoint up and skip the ones that are gone - #1098

Open
sirtimid wants to merge 4 commits into
sirtimid/terminate-vat-through-run-loopfrom
sirtimid/endpoint-lookup-and-skip
Open

sirtimid wants to merge 4 commits into
sirtimid/terminate-vat-through-run-loopfrom
sirtimid/endpoint-lookup-and-skip

Conversation

@sirtimid

@sirtimid sirtimid commented Sep 15, 2026

Copy link
Copy Markdown
Contributor

Stacked on the vat-lifecycle chain; base is sirtimid/terminate-vat-through-run-loop.
Its own diff is
git diff sirtimid/terminate-vat-through-run-loop...sirtimid/endpoint-lookup-and-skip.

A send whose endpoint is gone splats: its result promise is rejected and the
caller learns. A notify, a GC action and a reap have nobody to tell, and they
looked their endpoint up with a bare #getEndpoint, which throws. That throw
escapes the crank and kills the run loop — and because the crank is rolled back
and the item restored, it kills the next boot the same way. One bringOutYourDead
from a peer is enough: the kernel answers it by scheduling a reap in the
persisted reap queue, nothing purges that queue, and the next incarnation starts
its run loop inside Kernel.make, before an embedder can call initRemoteComms.

They now look the endpoint up and skip the ones that are gone, which is what
SwingSet does — vatWarehouse.lookup and "can't collect from the dead".

Gone means a vat with no handle. With a restart and a termination each
carried out inside a crank of its own, by the PRs below this one, no crank can
find a live vat between workers, so an absent handle is a vat that has ended. No
between-incarnations branch is carried over from #1023; there is no such state
left to have one for.

A remote with no handle has not gone anywhere. The kernel holds no remote at
all until the embedder calls initRemoteComms. Dropping a notify for one loses
a resolution its peer waits on forever — and because the incarnation id is
persisted, a plain restart is not an incarnation change that would sweep it. So
only a reap, which is a hint and will be asked again, may skip a remote. A
notify or a GC action addressed to one keeps today's loud failure. #1099, stacked
on this one, gives GC actions somewhere to wait — it aborts and keeps the action
rather than dropping it. A notify to an unreachable remote is left for a change
that can reorder the run queue itself. That is also what #1070 asks for: "handle absent" is not "remote gone".

Supersedes #1029 by @grypez, which fixed the same defect against main — it
added #getEndpointIfRunning over all three deliveries, kept the kernel's half
of a skipped GC action, and filtered krefs whose c-list entries a cleanup took
in the same crank. All three are carried here, along with its integration test
and its gc.test.ts reachability case. It treats a remote the same as a vat,
which is the one thing this does differently, for the reason above. Left open
for @grypez to close or to object here.

This also closes the hole flagged on #1088 (getImporters counts remotes and
terminated vats): shouldProcessAction gates retireImport on hasCListEntry
alone, while #deliverGCAction looked its endpoint up bare — so a GC action for
a terminated vat whose c-list cleanup has not reached it yet threw out of the
crank. That case is closed here; the remote case is closed by the PR stacked on
this one. #1088 is safe to merge once both land.

Changes

  • KernelRouter#lookupEndpoint, used by #deliverNotify, #deliverGCAction
    and #deliverBringOutYourDead. It keys on VatNotFoundError, so a fault that
    is not "this vat has ended" still throws.
  • A skipped GC action still performs the kernel's own half — the action is
    already spent from the durable set, and leaving the entry has it re-derived
    forever — over the krefs whose c-list entries survive, since a
    nextTerminatedVatCleanup in the same crank can take them and krefsToErefs
    reports a missing one by throwing.
  • #deliverNotify looks the endpoint up after releasing the queued
    notification's own reference and before the translations, which import if
    needed: minting c-list entries for an endpoint nobody will tell writes rows
    only that endpoint could release.
  • GC_DELIVERY replaces a method name assembled from the action type and cast
    back into range (draft A9).
  • @metamask/ocap-kernel changelog entry under Fixed.

Testing

KernelRouter.test.ts gains two describes: one for an endpoint named by
persisted state that is not running — notify, GC action and reap each skipped
rather than thrown, the kernel's half still done, nothing allocated in the
skipped endpoint's c-list, the notify's own reference still released, and an id
that names no endpoint still throwing — and one for a remote that is only out of
reach, where the reap goes and the three GC actions stay loud. Eight separate
mutations were checked, including the two that a first draft of these tests let
through: hoisting the notify's lookup above decrementRefCount, which leaks a
reference, and releasing every kref rather than the surviving ones.

store/methods/gc.test.ts pins the reachability the guard exists for: a reap
scheduled for a vat outlives the vat's termination and every cleanup after it.

kernel-test/src/remote-comms.test.ts carries #1029's integration test. A peer's
unsolicited bringOutYourDead persists a reap, and the victim is then booted
twice: both must come up running. Twice, because the failure is unrecoverable
rather than merely fatal — the crank that dies is rolled back, putting the reap
back for the boot after. Verified against this branch's code, not only Ryan's:
with the reap's skip turned off, both boots report
{ state: 'failed', error: 'Remote not found: r1' }.

Full @metamask/ocap-kernel suite green locally; eslint, constraints and
changelog:validate clean.

Closes #1061
Closes #1070

🤖 Generated with Claude Code


Note

High Risk
Changes core run-queue delivery and GC/notify semantics on missing vats and not-yet-restored remotes; incorrect skipping could leak refs or drop peer-visible work, though behavior is heavily tested.

Overview
Fixes a run-loop brick where notify, GC actions, and bringOutYourDead called #getEndpoint directly; a missing vat or remote at boot (e.g. a persisted reap before initRemoteComms) threw out of the crank, rolled back, and blocked every subsequent boot.

KernelRouter adds #lookupEndpoint: deliveries to a vat with no handle are skipped (warn + { didDelivery }) instead of killing the crank. For remotes with no handle yet, only bringOutYourDead may be dropped (discardable); notify and GC still throw so resolutions/releases are not silently lost. Skipped GC still runs the kernel-side c-list work on krefs that still have entries; notify lookup happens after releasing the queued ref and before translations that would mint c-list rows. GC delivery uses a GC_DELIVERY map instead of string-built method names.

Tests cover missing endpoints, remote-only-unreachable behavior, reap-after-vat-termination in the store, and an integration case where a peer’s unsolicited bringOutYourDead must not brick restart. Changelog updated under Fixed.

Reviewed by Cursor Bugbot for commit ce689f4. Bugbot is set up for automated code reviews on this repo. Configure here.

sirtimid and others added 4 commits September 16, 2026 18:26
A notify, a GC action and a reap have nobody to reject. Looking their
endpoint up bare meant that an endpoint named by persisted state but not
running threw out of the crank, which killed the run loop — and since
the crank was rolled back, the item came back and killed the next boot
too. A vat's c-lists outlive it until `cleanupTerminatedVat` reaches
them, one vat per crank, and no remote exists until `initRemoteComms`
runs, so one `bringOutYourDead` from a peer was enough to brick a kernel.

They now skip, per SwingSet's "can't collect from the dead". The
kernel's own half of a GC action still happens, because the action is
already spent from the durable set; where the endpoint is a remote the
kernel still has a record for, that leaves the peer holding references
this kernel has released, and it says so.

No between-incarnations case is carried over: with a restart and a
termination each carried out inside a crank of its own, a vat with no
handle is a vat that is gone.

Closes #1061
Closes #1070

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
… reach

Review follow-up. Two reviewers found the same hole: a remote with no
handle has not gone anywhere — the kernel holds none at all until the
embedder calls `initRemoteComms`, which is after the run loop starts —
so skipping a notify addressed to one dropped a promise resolution the
peer waits on forever, and a persisted incarnation id means a restart is
not an incarnation change that would sweep it.

Only a reap, which is a hint, may skip a remote. A notify or a GC action
to one keeps today's loud failure; deferring it needs a lane of its own.
`#lookupEndpoint` now keys on `VatNotFoundError` rather than on the shape
of the id, so a real fault is no longer swallowed, and the skip is a
warning rather than an error, being the ordinary end of a vat's life.

`hasRemoteInfo` goes with the distinction it was added for.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…s view

Lane C traced 13/14/15 and found two states that reach a crank with an
active vat record and no handle: a vat torn down after its stream died,
whose queued termination has not landed yet, and one whose launch failed
at startup and which nothing relaunches. Neither is marked terminated,
and the second never will be.

The skip already went by the missing handle, so this is a guard rather
than a fix — gating it on the store's view instead fails eleven tests
now, including one that says so directly. The comment claimed a
narrower reason than the truth and has been corrected.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@sirtimid
sirtimid force-pushed the sirtimid/endpoint-lookup-and-skip branch from 8a48a04 to ce689f4 Compare September 16, 2026 16:29

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

1 participant