fix(dbx): make LLM request timeouts configurable via environment - #274
Open
xw-song wants to merge 1 commit into
Open
fix(dbx): make LLM request timeouts configurable via environment#274xw-song wants to merge 1 commit into
xw-song wants to merge 1 commit into
Conversation
The non-streaming and streaming LLM timeouts were fixed at 300s and 600s. Generation throughput varies between serving endpoints, so a single reasoning-tier pass can exceed 300s and abort mid-run, surfacing as a FetchTimeoutError naming the endpoint rather than the real cause. Read both from FORGE_LLM_TIMEOUT_MS and FORGE_LLM_STREAM_TIMEOUT_MS, falling back to raised defaults of 900s and 1200s. Malformed or non-positive values fall back rather than throwing.
|
Codex usage limits have been reached for code reviews. Please check with the admins of this repo to increase the limits by adding credits. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
The LLM timeouts in
lib/dbx/model-serving.tsare fixed constants (300snon-streaming, 600s streaming). On a slower serving endpoint a single
reasoning-tier pass can exceed 300s and abort mid-run, surfacing as an error that
names the endpoint and reads like it is unreachable:
Measured on an Azure workspace,
databricks-claude-sonnet-4-6pay-per-token: a17K-token prompt returns in 6.9s, but sustained generation runs at ~41 output
tok/s —
max_tokens=1500took 36.7s, andmax_tokens=4000timed out at 321.7s.Time to first token is fine; long generations are the constraint.
This reads both timeouts from the environment so a deployment can raise them
without a code change:
FORGE_LLM_TIMEOUT_MS— default 900000 (was 300000)FORGE_LLM_STREAM_TIMEOUT_MS— default 1200000 (was 600000)Malformed, zero, negative and non-finite values fall back to the default rather
than throwing.
Type of Change
Pre-Merge Checklist
npm run lintpassesnpm run typecheckpassesnpm testpasses234 files / 2420 tests passing.
__tests__/model-serving/env-timeout.test.tscovers unset, valid, empty, non-numeric, zero, negative and
Infinity. Notdocumented in
docs/— happy to add it alongside the otherFORGE_*vars.Notes for Reviewers
Demo research already runs detached from the HTTP request
(
app/api/demo/research/route.tsfiresvoid startResearch()and the clientpolls for status), so a longer ceiling cannot trip a gateway timeout. This
constant was the only deadline in that path.
Trade-off: a wedged request now holds a rate-limiter slot for up to 15 minutes
instead of 5, which on a saturated pool could delay queued work.
envTimeoutMsis exported only so it can be unit-tested.