fix: Reap JobRunner subprocesses before os._exit to prevent orphans and semaphore leaks

[hongquanli]

Summary

Fixes two user-visible symptoms of the same bug: orphaned multiprocessing spawn_main processes (PPID 1) accumulating after GUI sessions, and the shutdown warning resource_tracker: There appear to be N leaked semaphore objects to clean up at shutdown.

Root cause

main_hcs.py exits via os._exit() (deliberate — PyQt5's C++ destructor order conflicts with Python's GC). But os._exit() also skips multiprocessing's atexit hook, the mechanism that normally terminates daemon child processes. Any JobRunner subprocess still alive at that moment is orphaned to PPID 1 and its queue/event semaphores leak.

The per-acquisition cleanup (_finish_jobs) shuts runners down in non-blocking daemon threads, with a comment claiming "if app exits before threads complete, OS will terminate subprocesses anyway" — untrue under os._exit(). Runners stay alive whenever their shutdown event was never set: the exit-time abort join times out (15 s cap), the worker dies before _finish_jobs, or the shutdown threads lose the race to os._exit().

Fix

• shutdown_all_job_runners() (job_processing.py): finds live JobRunner children via multiprocessing.active_children(), runs their full shutdown() in parallel (bounded), then terminates any survivor. Called in main_hcs.py immediately before os._exit().
• JobRunner.shutdown() now also clears _ready_event, releasing its semaphore like the other primitives.
• Corrected the _finish_jobs comment.

Testing

• New tests/control/core/test_job_runner_teardown.py: a runner whose shutdown event was never set is reaped (deterministic reproduction of the leak condition); no-op with no runners; safe after a prior clean shutdown.
• All 65 existing JobRunner-related tests pass (test_job_runner_shutdown/pending, backpressure, zarr writer, OME-TIFF saving).
• GUI driver verification in simulation (programmatic Qt driver replicating main_hcs.py's exact exit path): normal close after an acquisition and close mid-acquisition — zero orphaned children, no resource_tracker warning.
• Note: the common GUI close paths already cleaned up correctly in control runs (the existing non-blocking shutdown usually wins its race); this fix closes the timing/crash windows, which matches how the leaks were observed in practice — occasional, and hours old.

🤖 Generated with Claude Code

[hongquanli]

Review guide (small, independent, ready)

4 files, +99/−1, from master, CI green. ~10-minute review — no hardware needed.

What it fixes: orphaned multiprocessing spawn_main children (PPID 1) after a normal GUI close, and the resource_tracker: leaked semaphore objects warning — two symptoms of one cause.

Root cause: main_hcs.py exits via os._exit() (deliberate, to dodge a PyQt5/GC teardown segfault), but os._exit() also skips multiprocessing's atexit hook — the thing that normally terminates daemon children. Any JobRunner still alive at that instant (abort-join timed out, worker died before _finish_jobs, or the non-blocking shutdown threads lost the race) is orphaned and leaks its queue/event semaphores.

What to check (the whole review):

• shutdown_all_job_runners() in job_processing.py — finds live JobRunner children via multiprocessing.active_children(), shuts them down in parallel, terminates survivors; called in main_hcs.py right before os._exit().
• JobRunner.shutdown() now also clears _ready_event (releases its semaphore like the other primitives).
• The _finish_jobs comment that claimed "the OS will terminate subprocesses anyway" — corrected (untrue under os._exit).

Evidence: unit test reaps a runner whose shutdown event was never set (deterministic repro of the leak); GUI driver close (normal + mid-acquisition) shows zero orphans, no warning. Live corroboration since: a crashed-pytest run this week accumulated 264 orphaned JobRunner children that poisoned the whole multiprocessing test suite machine-wide until swept — same leak, different trigger.

🤖 Generated with Claude Code