fix(llm): release concurrency permits during retry backoff

[koriyoshi2041]

Problem

Providers run their retry loops inside the wrapper semaphore scope. A call therefore holds its per-operation and global concurrency permits while sleeping between attempts, which can starve unrelated operations when the global limit is small. Fixes #3130.

Fix

Permits now represent active upstream attempts instead of whole logical calls. Providers that own a retry loop declare supports_attempt_scoped_concurrency(); the wrapper then skips holding the semaphores and passes an attempt_context factory that the provider enters around each individual upstream request — including structured output, tool calls, streaming SSE consumption, and native Ollama structured requests. This applies to all real providers (OpenAI-compatible, OpenAI Responses, Anthropic, Gemini, LiteLLM, Codex, Claude Code, llama.cpp via delegation), per review. Permit acquisition remains centralized and keeps the per-operation-then-global order.

Worker-stage observability is preserved and extended (#3002): a call queued on a saturated semaphore keeps its .queued stage until the first post-acquire attempt=N stamp, and a failed attempt gains a .backoff suffix while the provider sleeps without permits — so "waiting for a permit", "request in flight", and "backing off" are all distinguishable in worker logs.

Tests

• End-to-end regression through the real OpenAI-compatible retry loop: an unrelated call completes while another call sleeps out its backoff, and attempt 2 reacquires the permit before hitting the upstream.
• Cancellation while queued on the global permit releases the already-acquired per-operation permit.
• Stage lifecycle: .queued → attempt=N (only once permits are held) → .backoff during retry sleep.
• Wrapper wiring, attempt-context forwarding (llama.cpp delegation, Responses provider), and stage-publication tests.
• uv run pytest tests/test_llm_per_op_concurrency.py tests/test_llamacpp_attempt_context.py tests/test_openai_compatible_response_hardening.py tests/test_openai_responses_provider.py tests/test_llm_strict_schema.py tests/test_llm_trace.py ... — all green; ruff + ty clean.

Risk

Retry classification, backoff calculation, attempt counts, tracing context, and usage aggregation are unchanged. The main behavioral change is that queued retries now compete fairly with other calls instead of retaining their previous slot — under saturation a retrying call's end-to-end latency can grow, which is the intended fairness trade-off.

[normalnormie]

Review summary

The attempt-scoped permit design addresses the reported starvation mechanism correctly. OpenAI-compatible network requests remain concurrency-limited, while retry backoff occurs outside the permit context. Retry classification, attempt counts, and delay calculations appear unchanged.

Observability regression

The worker-stage contract currently states that a call remains marked .queued until its concurrency permits have been acquired. In the proposed OpenAI-compatible path, the stage changes to the base/.attempt=N state before attempt_context() acquires those permits.

Consequently, a call waiting on either semaphore can appear to be an active provider attempt—the ambiguity previously addressed by #3002.

Suggested behavior:

1. Set .queued before each permit acquisition.
2. Acquire the per-operation and global permits.
3. Set .attempt=N only after both permits are held.
4. Execute the upstream request.

Minimal regression coverage

A compact parameterized test covering both call() and call_with_tools() should be sufficient. With per-operation and global limits enabled, it should verify that:

• an unrelated call completes while another call is backing off;
• both permits are released during backoff;
• the retry reacquires them before its next upstream attempt.

Separate tests for every retry exception and provider path do not appear necessary because those paths share the same attempt context.

CI

The Core LLM test job was skipped in the current workflow run. The new concurrency regression should execute in required CI before merge.

Overall, the implementation direction is sound. Preserving the queued-stage contract and covering both public call paths would close the remaining correctness gap without expanding the scope of the PR.

[nicoloboschi]

good catch, can you apply this to all llm providers?

[koriyoshi2041]

Updated the PR to scope concurrency permits to each upstream attempt across OpenAI-compatible, Anthropic, Gemini/Vertex, LiteLLM/Router, Claude Code, Codex, and the LlamaCpp delegate. Retry backoff, auth refresh, parsing, and validation now run outside the permits. Added deterministic coverage for backoff release, stage ordering, and LlamaCpp context forwarding. Local validation: 180 passed, 4 skipped; Ruff, formatting, ty, and repository pre-commit hooks passed.