Skip to content

fix: per-job EventLoopThread + timeout kill for parallel tools#1781

Open
redcomet168 wants to merge 1 commit into
agent0ai:mainfrom
redcomet168:fix/parallel-eventloop-singleton
Open

fix: per-job EventLoopThread + timeout kill for parallel tools#1781
redcomet168 wants to merge 1 commit into
agent0ai:mainfrom
redcomet168:fix/parallel-eventloop-singleton

Conversation

@redcomet168

Copy link
Copy Markdown

Problem

All parallel jobs share a single EventLoopThread singleton (THREAD_BACKGROUND in helpers/defer.py). Any blocking I/O in one worker freezes all concurrent jobs on that shared loop. Timed-out jobs are marked but never killed — they leak as zombie threads and file descriptors.

This is the root cause behind issues #1485, #1011, and #1248.

Fix

Single file changed: helpers/parallel_tools.py — 7 insertions, 3 deletions.

1. Per-job event loop threads

start_parallel_jobs: each job gets its own DeferredTask with a unique thread name instead of the shared singleton:

- task = DeferredTask(thread_name=THREAD_BACKGROUND)
+ task = DeferredTask(thread_name=f"parallel-{job.id}")

Each job runs on an isolated event loop, so blocking I/O in one worker no longer freezes the others.

2. Kill timed-out jobs

await_parallel_jobs: when the deadline is reached, timed-out jobs are now terminated instead of just marked:

for job in active:
    if job.deferred_task:
        job.deferred_task.kill(terminate_thread=True)
    _finish_job(job, "timeout", error="Job exceeded timeout and was killed.")

3 & 4. Thread cleanup on cleanup/cancel

cleanup_parallel_job and _cancel_job: use kill(terminate_thread=True) to ensure the event loop thread is properly cleaned up.

What This Does NOT Change

  • No changes to helpers/defer.py — the EventLoopThread singleton pattern remains untouched
  • No changes to agent.py or initialize.py
  • No changes to the DeferredTask class itself
  • No new dependencies

Verification

Tested in a live Agent Zero v2.5 Docker deployment with a plugin applying identical changes (monkey-patch approach). Results:

Test Result
2-job parallel call Both completed in 2.3s (was hanging for 300s)
Timeout enforcement (60s job, 10s timeout) Killed at 10.3s, correct timeout state
4-job concurrent stress test All completed independently, clean thread cleanup
Process thread count Stable — no thread/FD accumulation after repeated calls

Tradeoffs

Per-job threads create more threads during parallel execution (one per job instead of one shared). Each thread is cleaned up after job completion via terminate_thread=True. This is a minor resource tradeoff for eliminating the freeze-everything symptom.

Fixes #1485
Fixes #1011
Fixes #1248

All parallel jobs shared a single EventLoopThread singleton (THREAD_BACKGROUND),
so any blocking I/O in one worker froze all concurrent jobs until timeout.
Timed-out jobs were marked but never killed, leaking threads and FDs.

Changes (helpers/parallel_tools.py only):
- start_parallel_jobs: use per-job thread name instead of shared singleton
- await_parallel_jobs: kill timed-out jobs with terminate_thread=True
- cleanup_parallel_job: use terminate_thread=True for thread cleanup
- _cancel_job: use terminate_thread=True for thread cleanup

No changes to defer.py, agent.py, or initialization.

Fixes agent0ai#1485, agent0ai#1011, agent0ai#1248
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

1 participant