Skip to content

ateworker-gvisor: reuse the checkpoint image left by a cancelled attempt - #2491

Closed
Davanum Srinivas (dims) wants to merge 1 commit into
agent-substrate:mainfrom
dims:ateom-checkpoint-reuse
Closed

Davanum Srinivas (dims) wants to merge 1 commit into
agent-substrate:mainfrom
dims:ateom-checkpoint-reuse

Conversation

@dims

@dims Davanum Srinivas (dims) commented Oct 12, 2026 •

Copy link
Copy Markdown
Collaborator

A retried CheckpointWorkload ran runsc checkpoint against a sandbox the first attempt had already checkpointed and stopped, and crashed the actor.

  • Reuse the image when checkpoint.img exists and the pause container is gone or stopped.
  • cmdState returns the container's status.

The detached post-checkpoint teardown this PR carried landed with #1917.

go test -race ./cmd/ateworker-gvisor/... on linux, lint clean.

@dims
Davanum Srinivas (dims) marked this pull request as ready for review October 12, 2026 13:24
@dims
Davanum Srinivas (dims) force-pushed the ateom-checkpoint-reuse branch 3 times, most recently from 3613722 to 1184d85 Compare October 13, 2026 22:21
@ronlv10

Copy link
Copy Markdown
Collaborator

Julian Gutierrez Oschmann (@juli4n) who worked on Idempotency lately

@dims
Davanum Srinivas (dims) force-pushed the ateom-checkpoint-reuse branch 4 times, most recently from ba5c588 to 0c26755 Compare October 16, 2026 12:47
@dims Davanum Srinivas (dims) changed the title ateom: reuse the checkpoint image left by a cancelled attempt ateworker-gvisor: reuse the checkpoint image left by a cancelled attempt Oct 16, 2026
@dims
Davanum Srinivas (dims) force-pushed the ateom-checkpoint-reuse branch 2 times, most recently from 99ea6c1 to 7cce01a Compare October 18, 2026 21:03
@dims
Davanum Srinivas (dims) force-pushed the ateom-checkpoint-reuse branch 8 times, most recently from 1b7a59d to 75f0d2a Compare October 24, 2026 11:39
A retried CheckpointWorkload ran `runsc checkpoint` against a sandbox the
first attempt had already checkpointed and stopped, got exit 128 and
crashed the actor. The first attempt had been cancelled by its caller
during the teardown.

Reuse the image when it exists and the pause container no longer runs.
cmdState now returns the container's status.
@chw120

Copy link
Copy Markdown
Collaborator

Thanks Davanum Srinivas (@dims)! #3015 (merged) covers this case: when a retried CheckpointWorkload finds the workload gone, it answers from a completion marker that the worker writes only after all snapshot files are written. #2623 (merged) clears partial files before a new attempt, and since it runs first, this PR's check no longer finds the earlier image. Please see more details in #2836.

@dims

Copy link
Copy Markdown
Collaborator Author

Chenyi Wang (@chw120) thank you! one PR less for me to think about and rebase :)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants