Repository navigation
Self-hosted Embed: failed pause or checkpoint loses the sandbox #3658
Description
Activity
- changed the title
[-]Self-hosted Embed: failed rootfs diff fsync during Pause leaves the sandbox unaddressable with no tested recovery path (v0.16.202609130627-59497eb9134)[/-][+]Self-hosted Embed: sandbox lost when pause fails[/+]on Sep 23, 2026 - changed the title
[-]Self-hosted Embed: sandbox lost when pause fails[/-][+]Self-hosted Embed: failed pause or checkpoint loses the sandbox[/+]on Sep 23, 2026 - added a commit that references this issue
on Sep 24, 2026 Thanks for the detailed report.
This is the intended fail-closed behavior. A pause or checkpoint only produces a restorable snapshot once every artifact has been written and synced to disk. When the
fsyncof the rootfs diff fails withEIO, which usually happens when the underlying disk, controller, or filesystem is faulty, the kernel discards the unwritten pages, so the diff cannot be trusted; we discard the whole snapshot instead of leaving something restorable that may be corrupt. The checkpoint case ends the same way: the snapshot was never persisted, so its build is marked failed, and the original sandbox had already been stopped as part of taking it. The current default on a failed snapshot is to delete.Closing as working as intended. Closing #3666 and #3668 as well.
Thanks for the detailed report.
This is the intended fail-closed behavior. A pause or checkpoint only produces a restorable snapshot once every artifact has been written and synced to disk. When the
fsyncof the rootfs diff fails withEIO, which usually happens when the underlying disk, controller, or filesystem is faulty, the kernel discards the unwritten pages, so the diff cannot be trusted; we discard the whole snapshot instead of leaving something restorable that may be corrupt. The checkpoint case ends the same way: the snapshot was never persisted, so its build is marked failed, and the original sandbox had already been stopped as part of taking it. The current default on a failed snapshot is to delete.Closing as working as intended. Closing #3666 and #3668 as well.
Thanks — agreed that a corrupt snapshot must be discarded, and that's not in dispute. LocalDiffFile.CloseToDiff already removes the partial diff on an fsync failure, and neither PR tries to retain a possibly-corrupt snapshot.
The point I'd push back on is narrower: discarding the bad artifact and destroying the sandbox are two separate actions, and only the first is forced by the fault.
On the pause path the guest isn't "stopped as part of taking the snapshot" — pauseVM issues PatchVM{State: Paused}, a suspend. At the moment fsync fails, the VM is still frozen and intact; the failing fsync is on an exported diff copy, not the live overlay. Destroying
it is a choice made in the error path (stopSandboxAsync), not a consequence of the EIO. That's the only thing #3668 changes: on snapshot failure it resumes the already-suspended guest in place and returns FailedPrecondition, corrupt artifact still discarded. It's
flag-gated, default off — byte-identical to today unless a self-hoster opts in.I take the fail-closed philosophy seriously, and I think the fair question isn't "discard or keep the snapshot" (discard — you're right) but "when the guest is demonstrably still alive at failure time, is destroying it the safe default or the aggressive one?" For a
transient trigger like the reported hugepage ENOMEM, resuming in place reuses memory the guest already holds — it needs no fresh allocation, so it's lighter than destroy-and-recreate, not riskier.
On self-hosted Embed, a failed pause or checkpoint can stop the original sandbox and leave no snapshot the SDK can restore. Its files and process state become inaccessible.
Reproduce
Pause: disk sync failure
sandbox.pause(keep_memory=True).straceto makefsyncreturnEIOfor the rootfs diff file during the pause.Pause returns
500: Error pausing sandbox; the orchestrator reportsfailed to sync file: input/output error. Reproduced 2/2 times. Without the injected error, pause/resume preserves files and process memory.Fault injection command
Attach after the diff file appears and before it is synced (about 1–2 seconds). Replace the path with the actual diff file path; a watcher was used to catch this window.
The original incident followed repeated host
No space left on deviceerrors. Those logs do not establish the cause of the laterfsyncerror.Checkpoint: memory allocation failure
Exhaust the host's reserved hugepage memory pool, then call
create_snapshot(). Starting the replacement sandbox fails withmmap memfd: cannot allocate memory. Reproduced 1/1 time with real memory exhaustion on the same release binary. Normal checkpoint/restore works with memory available.Actual result
In both cases:
get_inforeturns “not found.”failed; restoring it returns404: template not found.The pause failure deletes the local snapshot artifacts. The checkpoint failure leaves local files, but they cannot be restored through the SDK.
Expected result
A failed pause or checkpoint should leave the sandbox recoverable: either the original sandbox remains available or a usable snapshot can be restored.
Environment
file:///var/lib/e2b/storage/templates)v0.16.202609130627-59497eb9134(unmodified release binary)v0.14.202609170000-908833e4c12; Python SDK2.51.0a065a4ddb3f2c6a4149634d9acb14b62f65839acRelated: #3659 covers a snapshot marked successful that crashes on restore.