Repository navigation
fix(recorder): keep late progress updates from rewriting the state being sealed - #158
Merged
Merged
Conversation
…ing sealed The worker writes the final capture-state.json and then seals the capture directory, inventorying that file. A non-terminal control transition that lands after staging still rewrote it: the status-drain thread's late "record.stopping" (finalizing) on a loaded host, or a stop caller's finalization timeout. A write during the seal changed an inventoried artifact, so stop returned capture_seal_failed (test_concurrent_repeated_stop_is_idempotent on test-macos). A write just before it sealed a stale finalizing state. Once the final state is staged, ignore non-terminal transitions; a terminal outcome such as a seal failure can still replace it. Add a deterministic regression test that delivers the late update mid-seal.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
tests/test_control.py::test_concurrent_repeated_stop_is_idempotentfails intermittently ontest-macos:CaptureControlError('Capture control failed: capture_seal_failed')recording_or_finalization_failedRoot cause
This is a real race in
Recorder, not just a slow test. A timeout bump would not fix it.record()returns,_run_recordwrites the finalcapture-state.json(_stage_completed_control_state). It then callsseal_capture, which inventories every file in the capture directory, including that file, and verifies the inventory._transition_controlonly refused updates once the in-memory phase was terminal. That phase becomescompleteonly after the seal. During the seal, any non-terminal transition still rewrotecapture-state.json:record.stopping→finalizing, which happens when that thread is scheduled late on a loaded runner, orfinalization_timeouttransition.capture_seal_failed. If it lands just before the seal, the capture seals a stalefinalizingstate.Reproduced deterministically: delivering the
finalizingupdate right afterbuild_artifact_manifestreturns gives exactlycapture_seal_failed.Fix
Once the final state is staged,
_transition_controlignores non-terminal transitions. That file now belongs to the seal. A terminal outcome, such as a seal failure marking the capturefailed, can still replace it. A stop caller that times out still gets itsfinalization_timeoutresponse; the timeout just no longer rewrites a sealed artifact.Test
Added
test_late_progress_update_cannot_rewrite_the_state_being_sealed. It runs an in-process recorder and delivers latefinalizingandfinalization_timeoutupdates in the middle of the seal.capture_seal_failed.verify_capture_artifactspasses, and the sealed state iscomplete.Locally on macOS:
tests/test_control.pypassed 5 runs in a row, and the full non-slow suite passed (871 passed).