fix: per-job EventLoopThread + timeout kill for parallel tools#1781
Open
redcomet168 wants to merge 1 commit into
Open
fix: per-job EventLoopThread + timeout kill for parallel tools#1781redcomet168 wants to merge 1 commit into
redcomet168 wants to merge 1 commit into
Conversation
All parallel jobs shared a single EventLoopThread singleton (THREAD_BACKGROUND), so any blocking I/O in one worker froze all concurrent jobs until timeout. Timed-out jobs were marked but never killed, leaking threads and FDs. Changes (helpers/parallel_tools.py only): - start_parallel_jobs: use per-job thread name instead of shared singleton - await_parallel_jobs: kill timed-out jobs with terminate_thread=True - cleanup_parallel_job: use terminate_thread=True for thread cleanup - _cancel_job: use terminate_thread=True for thread cleanup No changes to defer.py, agent.py, or initialization. Fixes agent0ai#1485, agent0ai#1011, agent0ai#1248
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
All parallel jobs share a single
EventLoopThreadsingleton (THREAD_BACKGROUNDinhelpers/defer.py). Any blocking I/O in one worker freezes all concurrent jobs on that shared loop. Timed-out jobs are marked but never killed — they leak as zombie threads and file descriptors.This is the root cause behind issues #1485, #1011, and #1248.
Fix
Single file changed:
helpers/parallel_tools.py— 7 insertions, 3 deletions.1. Per-job event loop threads
start_parallel_jobs: each job gets its ownDeferredTaskwith a unique thread name instead of the shared singleton:Each job runs on an isolated event loop, so blocking I/O in one worker no longer freezes the others.
2. Kill timed-out jobs
await_parallel_jobs: when the deadline is reached, timed-out jobs are now terminated instead of just marked:3 & 4. Thread cleanup on cleanup/cancel
cleanup_parallel_joband_cancel_job: usekill(terminate_thread=True)to ensure the event loop thread is properly cleaned up.What This Does NOT Change
helpers/defer.py— theEventLoopThreadsingleton pattern remains untouchedagent.pyorinitialize.pyDeferredTaskclass itselfVerification
Tested in a live Agent Zero v2.5 Docker deployment with a plugin applying identical changes (monkey-patch approach). Results:
timeoutstateTradeoffs
Per-job threads create more threads during parallel execution (one per job instead of one shared). Each thread is cleaned up after job completion via
terminate_thread=True. This is a minor resource tradeoff for eliminating the freeze-everything symptom.Fixes #1485
Fixes #1011
Fixes #1248