Don't let a stale job persist overwrite a completed scan - #20
Draft
cursor[bot] wants to merge 1 commit into
Draft
cursor[bot] wants to merge 1 commit into
cursor[bot] wants to merge 1 commit into
Conversation
persist_job snapshotted ScanJob.to_dict() at schedule time and wrote it
asynchronously. A slower in-flight write of a 'running' snapshot could
finish after the terminal persist, leaving status=running on disk.
Reload then marked the finished scan as failed ('interrupted by server
restart'). Snapshot at write time and serialise per job so the latest
state always wins; drop the file if the job was deleted mid-write.
Co-authored-by: dmitryflynn <dmitryflynn@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
persist_job snapshotted job state at schedule time and wrote it asynchronously. A slower in-flight write of a
runningsnapshot could finish after the terminal persist, leavingstatus=runningon disk. On reload,ScanJob.from_dictthen marked the finished scan as failed ("Scan interrupted by server restart"), dropping the completed result.Trigger: a scan emits enough events to trigger a periodic persist, then completes while that write is still in the thread pool (JSON dump + fsync). Last writer wins with the stale snapshot.
Fix: snapshot at write time and serialize snapshot+write per job so the latest in-memory state always wins. If the job was deleted mid-write, drop the durable record so it is not resurrected.
Validation:
test_jobs_sse.py::TestPersistDoesNotClobberTerminalState(overlapping persist keepscompleted; delete during persist does not resurrect).python3 -m pytest test_jobs_sse.py— 27 passed.