fix(ocap-kernel): keep the kernel's account of a vat's life in one piece

[sirtimid]

Stacked on #1022, which is stacked on #1021; review those first. #1020 has merged. Replaces #1019, rebased onto the split stack.

Last of four. Closes #1015.

The defect

Only kernelStore.deleteVat() removes vatConfig.<vatId>. cleanupTerminatedVat sweeps ${vatId}.-prefixed keys, which never match vatConfig.<vatId>. So the two writes that together mean "this vat is dead" — deleteVat and markVatAsTerminated — both had to land, and nothing made them.

They were spread across four functions with awaits between them: VatManager.stopVat, #endVat's finally, VatHandle.terminate, and a lambda in Kernel.ts. A throw part-way left the vat marked terminated with its config alive. That reads as active again the moment cleanupTerminatedVat calls forgetTerminatedVat, at which point KernelRouter.#resolveEndpoint sees a vat the store calls live and the kernel has no handle for, rethrows, and kills the run loop. initializeAllVats then resurrects the vat on the next process start.

Two more, found alongside it:

• A vat whose stream failed tore itself down. VatHandle.#init's drain catch called this.terminate(true, …), which did deleteVat but never markVatAsTerminated and never removed the handle from VatManager.#vats. The router went on resolving the handle successfully, the write went nowhere, and since the vat RPC client has no timeout the crank never completed — the run loop hung while getRunLoopStatus() still reported running.
• A vat reported its dropped imports a bringOutYourDead late. makeGCAndFinalize called gc() without draining the queues first. A pending continuation still holds its closure's objects, so a sweep with work outstanding finds them reachable and the vat reports nothing on the BOYD that provoked it.

Approach

VatManager.#retireVat — the store side of a vat's death, in one synchronous step. Reject the promises it was deciding, unpin its root, deleteVat, markVatAsTerminated, with no await between them, so the half-written state cannot arise. Modelled on SwingSet's terminateVat (kernel.js:345-412) and its comment at :348 about the "synchronous prelude". Killing the worker is deliberately not part of it: that can fail, and a store that says the vat is dead is worth more than a store still waiting to find out.

#endVat and #abandonVat were both partial copies of this and are gone. Kernel.ts's lambda and launchVat lose their trailing markVatAsTerminated. VatHandle.terminate is left with only its own channel to close, and rejects its pending RPCs ahead of ending the stream rather than after, so a stream that will not close does not strand callers on a worker that is already dead.

stopVat(vatId, true, …) tolerates a missing handle. A restart needs a live handle to read its config from; an ending vat does not, and must not — the vat may be one the store still lists while the kernel has lost its handle, which is exactly what SubclusterManager.terminateSubcluster can hand it.

A fatal stream error is routed through the manager, via a new required onCriticalFailure prop on VatHandle. Only the manager can put the vat's death on record and drop the handle.

The run loop carries out a restart itself, as a queued restartVat item, so a vat is never out of the kernel's reach while cranks run.

#1015 is closed here, both halves: getImporters now counts remotes and terminated vats cleanup has not reached, so retiring an object queues a retireImport for each rather than deleting the object and leaving their c-list entries naming nothing. #1022 supplied the exemption site and the dangling discriminant; this supplies the missing importers. The vat half was raised by @grypez in review — see below.

Two conflicts the rebase surfaced, resolved on the merits

Both were real disagreements between this branch and the stack beneath it, not textual noise. They are in their own commit, 363c3e103.

maybeFreeKrefs was being cleared, not restored. #1021's revertStateBeneathRollback empties the set on rollback; this branch restores the savepoint's snapshot. Restoring is correct, and #1021's own comment admits the caveat ("correct only while every rollback discards the whole delivery"): the set is not per-crank — only collectGarbage empties it — so a candidate added while the run loop was idle, which terminateVat unpinning a root produces, was owed a collection and lost it to an unrelated crank's rollback. This branch introduces exactly that path. #1021's unit test had encoded the clear() behaviour and now distinguishes a pre-savepoint kref from one the abandoned crank added.

A fix from #1022 was regressed. #1022 marks a vat terminated unconditionally after a failed launch; this branch relied on stopVat reaching #retireVat to record it. But stopVat refuses a vat the kernel has no handle for and the store does not call active — which is what a partial launch looks like. The unconditional mark is asserted again.

Three fixes from review

Bugbot flagged four things on this branch; three are fixed here and the fourth is argued below.

A vat's death could not be recorded twice, and one path recorded it twice. deleteVat removes the vat's subcluster mapping, and removeVatFromSubcluster fails on a vat that has none, so the second call threw. performVatRestart reaches that whenever a relaunch breaks the stream: onCriticalFailure retires the vat, runVat rethrows, and the catch retires it again — so the throw escaped the one catch in this PR that must not throw, rolling back the termination records and returning the failing restart to the queue to be replayed forever. #retireVat is now a no-op the second time. Invisible to the suite until now, because the mock store's deleteVat was a bare vi.fn(); it refuses a repeat the way the real one does.

Two concurrent restart requests queued two items and kept one waiter. The first crank consumed the waiter and handed its caller a live handle; the leftover item then stopped that worker and, on a failed relaunch, retired the vat the caller had just been told about. A request arriving while one is still queued now takes over its item. The existing supersede test could not catch it — it stubs enqueueRestartVat to a no-op and drives performVatRestart once.

getImporters could not see a terminated vat still holding its imports. @grypez's finding, verified by execution; their repro reproduces byte for byte and their suggested fix is the one taken. Details in the review thread.

Both restart fixes are mutation-verified: revert the production hunk and the named test fails for the stated reason.

The fourth: a rolled-back crank can un-record a vat's death

Not fixed here, and I do not think it should be without a decision. onCriticalFailure drops the handle from #vats — plain RAM — and records the death in the store. Those store writes land inside the delivery savepoint, so a delivery that then throws rolls them back while the handle stays gone. Confirmed against a real SQLite store: after rollbackCrank('delivery') the store reports isVatActive: true, isVatTerminated: false, which is the store-says-live, kernel-has-no-handle disagreement this PR exists to prevent, and #resolveEndpoint turns it into a dead run loop.

#processCrankResult already has the mechanism this wants, and says so at the call site: crankResult.terminate runs after the rollback, inside crank, because "its writes must outlive the rollback above". Routing a mid-delivery stream death through that rather than writing the store from the callback is the fix I would argue for — but it reshapes the boundary between a delivery and the retirement that discovered it, and getting it wrong reintroduces the hang this PR removes. Worth its own change.

Testing

yarn lint clean, yarn build 31/31, changelog:validate clean. @metamask/ocap-kernel and @ocap/kernel-test fully green, with auditRefCounts on for every kernel kernel-test builds.

Note for reviewers of #1021: garbage-collection.test.ts › an object shared by two importers › survives until both importers let go was intermittently flaky on the branches beneath this one. The makeGCAndFinalize fix here is what addresses it.

Checklist

• I've updated the test suite for new or updated code as appropriate
• I've updated documentation (JSDoc, README.md, CHANGELOG.md) as appropriate

---

Note

High Risk
Changes core run-loop delivery, vat lifecycle/teardown, and store/crank boundaries; mistakes can resurrect vats, hang cranks, or kill the run loop on store/kernel disagreement.

Overview
Unifies vat death into a single synchronous #retireVat path (reject decider promises, unpin root, deleteVat, markVatAsTerminated) so partial termination cannot leave vatConfig alive while the vat is marked dead. Termination runs under #trackFlux / withStoreOutOfCrank instead of interleaved awaits; fatal stream errors go through VatHandle.onCriticalFailure to the manager, with optional reassertion out of crank if the death was rolled back with the handle already dropped.

Moves vat restart onto the run queue (restartVat items, enqueueRestartVat, performVatRestart) so no crank sees a vat between workers. Callers wait via #awaitRestart and KernelQueue.onRunLoopDeath when the loop dies first. Endpoint lookup is async (provideVat) and KernelRouter.#resolveEndpoint only drops work when a vat/remote is gone for good—not when the store still considers the vat active.

Hardens delivery and GC routing: result promises are rejected only if still unresolved; notify/BOYD/GC skip or propagate lookup failures appropriately; GC re-reads the c-list after awaiting the endpoint. makeGCAndFinalize drains microtasks before sweeping so BOYD reports dropped imports on the right round.

Supporting changes include refusing releaseSavepoint inside a crank, tests for transaction _spStack after failed rollbacks, and expanded integration tests for crank rollback vs termination/restart.

^{Reviewed by Cursor Bugbot for commit 2005001. Bugbot is set up for automated code reviews on this repo. Configure here.}

[github-actions]

Coverage Report

Status	Category	Percentage	Covered / Total
🔵	Lines	72.72% ⬆️ +0.67%	9665 / 13289
🔵	Statements	72.56% ⬆️ +0.67%	9821 / 13534
🔵	Functions	73.17% ⬆️ +0.30%	2270 / 3102
🔵	Branches	66.78% ⬆️ +0.98%	3943 / 5904

File Coverage

File	Stmts	Branches	Functions	Lines	Uncovered Lines
Changed Files
packages/ocap-kernel/src/Kernel.ts	89.84% ⬆️ +0.08%	79.54% ⬆️ +0.97%	85.41% 🟰 ±0%	89.84% ⬆️ +0.08%	332-334, 405, 429, 504-514, 602, 676, 752-755, 768, 778-779, 832, 855
packages/ocap-kernel/src/KernelQueue.ts	98.02% ⬇️ -0.54%	89.47% ⬇️ -0.80%	100% 🟰 ±0%	98.02% ⬇️ -0.54%	152, 202, 582
packages/ocap-kernel/src/KernelRouter.ts	94.73% ⬆️ +0.80%	85.26% ⬆️ +6.80%	100% 🟰 ±0%	94.7% ⬆️ +0.77%	127, 190, 207, 282, 337, 397, 424, 427, 541
packages/ocap-kernel/src/types.ts	100% 🟰 ±0%	100% 🟰 ±0%	100% 🟰 ±0%	100% 🟰 ±0%
packages/ocap-kernel/src/garbage-collection/gc-finalize.ts	77.27% ⬆️ +2.27%	75% 🟰 ±0%	100% 🟰 ±0%	77.27% ⬆️ +2.27%	27-31, 72-76
packages/ocap-kernel/src/store/types.ts	100% 🟰 ±0%	100% 🟰 ±0%	100% 🟰 ±0%	100% 🟰 ±0%
packages/ocap-kernel/src/store/methods/crank.ts	98% ⬇️ -2.00%	88.88% ⬇️ -4.87%	100% 🟰 ±0%	98% ⬇️ -2.00%	126
packages/ocap-kernel/src/store/methods/remote.ts	98.48% ⬆️ +0.05%	100% 🟰 ±0%	100% 🟰 ±0%	98.48% ⬆️ +0.05%	105-109
packages/ocap-kernel/src/store/methods/vat.ts	98.44% ⬆️ +1.15%	89.47% ⬆️ +7.66%	100% 🟰 ±0%	98.43% ⬆️ +1.16%	298-299
packages/ocap-kernel/src/vats/VatHandle.ts	90.76% ⬆️ +0.62%	86.66% 🟰 ±0%	100% 🟰 ±0%	90.76% ⬆️ +0.62%	383-388, 397-403
packages/ocap-kernel/src/vats/VatManager.ts	97.98% ⬇️ -2.02%	97.5% ⬇️ -2.50%	91.17% ⬇️ -8.83%	97.97% ⬇️ -2.03%	358-360, 392-403

Generated in workflow #4648 for commit c9b917b by the Vitest Coverage Report Action

[cursor]

Stale Bugbot comment from a previous run.

[cursor]

Stale Bugbot comment from a previous run.

[cursor]

Stale Bugbot comment from a previous run.

[cursor]

Cursor Bugbot has reviewed your changes using high effort and found 1 potential issue.

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.}

^{Reviewed by Cursor Bugbot for commit 2005001. Configure here.}

[cursor]

Partial retire is not idempotent

Medium Severity

#retireVat treats isVatTerminated as “already done”, but that mark is the last write. A first call that fails after resolvePromises or deleteVat leaves the flag unset, so a later call repeats those steps. resolvePromises throws on a settled promise and deleteVat throws once the subcluster mapping is gone. performVatRestart lets that throw escape the catch that is supposed to keep a failed restart from being rolled back and replayed.

Additional Locations (1)

• packages/ocap-kernel/src/vats/VatManager.ts#L610-L627

 

^{Reviewed by Cursor Bugbot for commit 2005001. Configure here.}

[sirtimid]

Converting to draft. This stack is being split into small, single-purpose PRs against main rather than reviewed as a whole; it stays open as the reference to cherry-pick from and will be closed once its last piece has landed. Please do not spend review time on it in this form.

[sirtimid]

Closing: this is now fully re-landed as individually reviewed PRs against main.

The split is described in the plan agreed on 2026-09-15; the vat-lifecycle concurrency this PR fought is answered in the accompanying decision record, only the run loop writes the store. So the pieces below are rewritten to that shape rather than cherry-picked: #vatsInFlux, #trackFlux, provideVat and the async #getEndpoint exist in none of them, because with restart and terminate on the run queue there is no window for them to paper over.

What this PR became:

piece	PR
makeGCAndFinalize drains the queues before gc(); reapAndSettle polls	#1083
#rejectResultIfPending: a settled result promise is not re-rejected from the delivery catch	#1084
#retireVat: a vat's death recorded in one synchronous step; stopVat and launchVat teardown	#1093
Restart is a run-queue item, with a per-vat waiter array and stale items dropped at startup	#1096
Control-plane terminateVat goes through the run loop	#1097
Stream death routed through the manager, guarded by handle identity	#1100
Endpoint resolution is lookup-and-skip	#1098

Each is motivated on its own, reproduces on main where it is a defect, and carries a test named for the behaviour. Nothing here is lost; the branch stays for reference.