Skip to content

Don't strand assigned scans after a controller restart - #28

Draft
cursor[bot] wants to merge 1 commit into
mainfrom
cursor/critical-bug-management-5fd1
Draft

cursor[bot] wants to merge 1 commit into
mainfrom
cursor/critical-bug-management-5fd1

Conversation

@cursor

@cursor cursor Bot commented Sep 1, 2026

Copy link
Copy Markdown
Contributor

Bug and impact

A queued scan that has already been assigned to a remote agent never runs if the controller restarts before the agent polls /tasks.

Trigger: Submit a scan; dispatcher sets assigned_agent_id and pushes the job onto the agent's in-memory pending_tasks. Controller crashes or is deployed before the agent polls. Agent comes back online.

Impact: The job stays queued forever in the UI/DB. try_dispatch_queued only considers unassigned jobs; reclaim_stale_jobs skips agents that are still online. Graceful shutdown previously also marked those waiting jobs failed ("Scan interrupted by server shutdown"), so every SaaS deploy aborted scans that had not even started.

This is distinct from reclaiming work from a dead agent (PR #22): here the agent is alive and the assignment survived on disk, but the process-local queue did not.

Root cause

Agent.pending_tasks is intentionally not persisted. After restart the durable job still has assigned_agent_id, so dispatch will not reassign it, and reclaim will not unstick it while the agent heartbeats.

Fix

  • Restore assigned queued jobs onto the agent's pending queue when it heartbeats or polls for tasks (before dispatching new work, so concurrency accounting is correct).
  • On shutdown, only fail running jobs (in-process scan threads die with the controller). Leave queued jobs for the assigned agent to pick up after check-in.

Validation

python3 -m pytest test_agent_architecture.py::TestRehydrateAssignedQueue test_production_readiness.py::TestAgentLifecycle::test_task_poll_recovers_assignment_after_pending_queue_loss — 7 passed.
Related suites (test_agent_architecture, test_production_readiness, test_jobmanager_concurrency, test_jobs_sse, test_pg_jobs, test_api_advanced) — 102 passed.

Open in Web View Automation 

pending_tasks is not persisted, so a crash or restart left queued jobs
pinned to a still-online agent that dispatch and reclaim both skip.
Restore that queue when the agent checks in, and stop failing queued
jobs on shutdown so they can be picked up after deploy.

Co-authored-by: dmitryflynn <dmitryflynn@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant