Don't strand assigned scans after a controller restart - #28
Draft
cursor[bot] wants to merge 1 commit into
Draft
cursor[bot] wants to merge 1 commit into
cursor[bot] wants to merge 1 commit into
Conversation
pending_tasks is not persisted, so a crash or restart left queued jobs pinned to a still-online agent that dispatch and reclaim both skip. Restore that queue when the agent checks in, and stop failing queued jobs on shutdown so they can be picked up after deploy. Co-authored-by: dmitryflynn <dmitryflynn@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Bug and impact
A queued scan that has already been assigned to a remote agent never runs if the controller restarts before the agent polls
/tasks.Trigger: Submit a scan; dispatcher sets
assigned_agent_idand pushes the job onto the agent's in-memorypending_tasks. Controller crashes or is deployed before the agent polls. Agent comes back online.Impact: The job stays
queuedforever in the UI/DB.try_dispatch_queuedonly considers unassigned jobs;reclaim_stale_jobsskips agents that are still online. Graceful shutdown previously also marked those waiting jobsfailed("Scan interrupted by server shutdown"), so every SaaS deploy aborted scans that had not even started.This is distinct from reclaiming work from a dead agent (PR #22): here the agent is alive and the assignment survived on disk, but the process-local queue did not.
Root cause
Agent.pending_tasksis intentionally not persisted. After restart the durable job still hasassigned_agent_id, so dispatch will not reassign it, and reclaim will not unstick it while the agent heartbeats.Fix
Validation
python3 -m pytest test_agent_architecture.py::TestRehydrateAssignedQueue test_production_readiness.py::TestAgentLifecycle::test_task_poll_recovers_assignment_after_pending_queue_loss— 7 passed.Related suites (
test_agent_architecture,test_production_readiness,test_jobmanager_concurrency,test_jobs_sse,test_pg_jobs,test_api_advanced) — 102 passed.