feat(worker): make the graceful-shutdown timeout configurable - #3092
Open
ai-ag2026 wants to merge 1 commit into
Open
feat(worker): make the graceful-shutdown timeout configurable#3092ai-ag2026 wants to merge 1 commit into
ai-ag2026 wants to merge 1 commit into
Conversation
The worker poller's graceful-shutdown timeout is hardcoded to 30s in both shutdown paths (worker/main.py and the API-embedded poller in api/http.py). An in-flight retain is an LLM call that may legitimately run for minutes — HINDSIGHT_API_LLM_TIMEOUT defaults far above 30s — so every service stop cancels it mid-flight and the operation is lost. Add HINDSIGHT_API_SHUTDOWN_GRACE (default 30.0 = unchanged behaviour), wired through HindsightConfig like the other operational knobs, and use it at both call sites. Deployments size it together with their supervisor stop timeout (e.g. systemd TimeoutStopSec) so the supervisor never SIGKILLs mid-cleanup. Tests: config default + env parsing, in the pattern of the neighbouring config-wiring test files.
nicoloboschi
requested changes
Aug 3, 2026
nicoloboschi
left a comment
nicoloboschi
left a comment
Collaborator
There was a problem hiding this comment.
please add the new field to the docs
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
The worker poller's graceful-shutdown timeout is hardcoded to
30.0in bothshutdown paths (
worker/main.py, and the API-embedded poller inapi/http.py). An in-flight retain is an LLM call that may legitimately runfor minutes —
HINDSIGHT_API_LLM_TIMEOUTdefaults far above 30s — so everyservice stop cancels it mid-flight and loses the operation.
This adds
HINDSIGHT_API_SHUTDOWN_GRACE(default30.0, behaviourunchanged unless set), wired through
HindsightConfiglike the otheroperational knobs, and used at both call sites.
Why
Running Hindsight under systemd, we size this together with
TimeoutStopSecso the supervisor never SIGKILLs the process mid-cleanup:grace comfortably under the supervisor timeout, both above the realistic
LLM-call duration. In production since 2026-07-16; restarts stopped eating
in-flight retains.
Changes
config.py:ENV_SHUTDOWN_GRACE+DEFAULT_SHUTDOWN_GRACE+HindsightConfig.shutdown_grace+from_envparsingworker/main.py,api/http.py:shutdown_graceful(timeout=get_config().shutdown_grace).env.exampledocumentedValidation
tests/test_shutdown_grace_config.py: default-is-30 and env override, inthe pattern of the neighbouring config-wiring test files (2 passed).