fix: progress-based job drain at acquisition end + JobRunner death detection

[hongquanli]

Summary

• No more lost images at acquisition end. _finish_jobs used a fixed 10 s deadline to wait for pending save jobs on every acquisition end (natural completion AND abort), then killed the runners. Under backpressure saturation the pending count equals the job limit by construction, so whenever pending × per-job save time exceeded 10 s the tail of the acquisition was abandoned and its images silently lost — even while jobs were completing steadily. The drain deadline is now progress-based: every completed job resets the stall clock, so a full-but-draining queue gets as long as it needs; jobs are abandoned only when nothing completes for stall_timeout_s.
• Dead JobRunner subprocess is now detected. Backpressure counters are only decremented inside the subprocess, so its death froze them at the limit: the acquisition crawled at one 30 s throttle timeout per frame, presenting as "stuck with jobs queue full", with every dispatched image lost. The worker now detects a dead runner with pending jobs at the throttle site (before waiting and after a capacity timeout) and aborts with a clear error.
• Abandonment is surfaced. Any abandoned jobs increment _acquisition_error_count, so the run ends as completed_with_errors instead of reporting clean success.

Evidence

Observed 2026-07-05 (headless run, simulation): queue pinned at jobs=20/20 (MB=662.7/10000 — the byte budget was at 6.6 %; the job cap was the binding limit), abort requested, 17 jobs drained in the 10 s window at ~0.55 s each, ~20 were pending → Timed out after 10 [s] waiting for jobs to finish. Pending jobs for SaveImageJob abandoned!!!. Full investigation write-up: AI-docs Squid/to-do/2026-07-12-backpressure-drain-findings.md.

Changes

• control/core/job_processing.py: new module-level drain_runners() (progress-based drain; kills a runner only on stall; abandons a dead runner's jobs immediately), find_dead_runners(), DrainResult; new JobRunner.pending_count() (has_pending() delegates to it and is now safe after shutdown() instead of raising on the cleared counter).
• control/core/multi_point_worker.py: _finish_jobs uses drain_runners (timeout_s → stall_timeout_s), logs abandonment cause per job class, counts abandoned jobs into _acquisition_error_count; new _abort_if_job_runners_dead() checked at the backpressure throttle site; background runner shutdown uses a short fixed join timeout (runners are idle after a clean drain, killed otherwise).

Test plan

• TDD: 10 new tests in tests/control/core/test_job_drain.py written first and watched fail (headline regression: 6 × 0.4 s jobs drain fully with stall_timeout_s=1.0 — the old fixed deadline would have abandoned them; stall abandonment; dead-runner immediate abandonment; find_dead_runners cases; pending_count lifecycle)
• pytest tests/control/core — 331 passed, 2 skipped
• black --config pyproject.toml clean

Notes

• Complements open fix: detect JobRunner subprocess death and stop memory-profiler ESRCH spam #526 (JobRunner death detection for memory-profiler ESRCH spam): this PR adds death detection in the acquisition path itself.
• Follow-ups deliberately not in this PR (see findings doc): let the byte budget be the primary throttle instead of the job-count cap; surface backpressure stats in headless/service runs.

🤖 Generated with Claude Code