feat(context): meter final requests and recover safely - #3850
Conversation
There was a problem hiding this comment.
🟡 Changes recommended
Monitoring evidence currently isn’t actually allowlisted/sanitized (risking sensitive data leakage) and tool-call streaming assembly can duplicate IDs/names when providers repeat fields across chunks.
Once you've addressed the issues Copilot identified, you can request another Copilot review.
Pull request overview
This PR adds “final request” metering and recovery to the SDK model layer, so budgeting decisions are based on the fully-rendered provider request (including tools/media/protocol overhead), and recovery/continuations are driven by provider-observed outcomes rather than local heuristics. It also enriches monitoring with content-free evidence and normalized per-call usage records.
Changes:
- Introduces a final-request metering + calibration system (
final_request_budget) and a runtime provider counting path (provider_request_count) to estimate/verify the physical request size. - Adds normalized, content-free provider usage recording (
provider_usage) and wires it intoOpenAIModelstreaming, retries, overflow recovery, and length continuations. - Extends monitoring/event emission for final-request evidence and context budget events; adds extensive test coverage across these flows.
File summaries
| File | Description |
|---|---|
| test/sdk/monitor/test_monitoring.py | Updates expected budget snapshot field value (uncertainty_reserve_basis). |
| test/sdk/monitor/test_context_budget_evidence.py | Adds tests for final-request evidence enrichment + allowlisting expectations. |
| test/sdk/core/models/test_provider_usage.py | Adds tests for provider usage normalization and schema contract. |
| test/sdk/core/models/test_provider_request_count.py | Adds tests for runtime provider-side input token counting and SSRF/shape gating. |
| test/sdk/core/models/test_openai_llm.py | Expands tests for provider usage records, tool-call streaming assembly, max_tokens rules, overflow recovery, and length continuation. |
| test/sdk/core/models/test_final_request_budget.py | Adds tests for request shape metering, calibration behavior, and recovery guards. |
| test/sdk/core/agents/test_context_composition.py | Adds tests for context composition segmentation + reconciliation behavior. |
| test/sdk/core/agents/test_context_budget_event.py | Adds tests ensuring budget events are content-free and allowlist compression reasons. |
| sdk/nexent/monitor/monitoring.py | Adds ContextVar-backed final-request evidence and enriches monitoring records with it. |
| sdk/nexent/core/models/provider_usage.py | Adds normalization utilities and a per-call usage record model. |
| sdk/nexent/core/models/provider_request_count.py | Adds runtime provider count client with strict capability + SSRF safeguards. |
| sdk/nexent/core/models/openai_llm.py | Integrates final-request metering, provider usage recording, overflow recovery, and length continuation into the model adapter. |
| sdk/nexent/core/models/final_request_budget.py | Implements final-request shape metering, calibration store, and safety guards for recovery. |
| sdk/nexent/core/agents/context/composition.py | Adds content-free context segment estimation + reconciliation to provider denominators. |
| sdk/nexent/core/agents/context_budget_event.py | Adds content-free budget event construction with allowlisted compression reasons. |
Review details
Suppressed comments (1)
sdk/nexent/core/models/openai_llm.py:693
- Tool-call function names are accumulated with
+=, which can duplicate the name if a provider repeats it across chunks. Names should be set once (first non-empty), while arguments can continue to be concatenated.
arguments = getattr(function, "arguments", None)
if name:
fragment["name"] += str(name)
if arguments:
fragment["arguments"] += str(arguments)
- Files reviewed: 15/15 changed files
- Comments generated: 3
- Review effort level: Lite
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
| def _enrich_record_with_final_request_evidence(record: Dict[str, Any]) -> None: | ||
| evidence = dict(_monitoring_final_request_evidence.get() or {}) | ||
| if not evidence: | ||
| return | ||
| prompt_usage = record.get("input_tokens") | ||
| if isinstance(prompt_usage, int) and prompt_usage > 0: | ||
| evidence["provider_prompt_usage_tokens"] = prompt_usage | ||
| error_text = str(record.get("error_message") or "").lower() | ||
| if any(marker in error_text for marker in ( | ||
| "context_length_exceeded", "maximum context length", "input tokens exceed", | ||
| )): | ||
| evidence["provider_overflow"] = True | ||
| record["context_budget_evidence"] = evidence |
| call_id = getattr(tool_call, "id", None) | ||
| call_type = getattr(tool_call, "type", None) | ||
| function = getattr(tool_call, "function", None) | ||
| if call_id: | ||
| fragment["id"] += str(call_id) | ||
| if call_type: | ||
| fragment["type"] = str(call_type) |
| def _valid_estimate(value: Any) -> Optional[int]: | ||
| parsed = _coerce_non_negative_int(value) | ||
| return parsed if parsed is not None else None | ||
|
|
4d5294f to
4e92cb2
Compare
4e92cb2 to
3a7cff4
Compare
3a7cff4 to
35aac9f
Compare
35aac9f to
870634c
Compare
929a8fc to
dfc98d1
Compare
|
补充 2026-09-03 变更：OpenAI adapter 在最终请求发出前应用统一有效策略：effort 型 reasoning 映射为 reasoning_effort，toggle 型映射为 extra_body.enable_thinking，并保留无关 extra_body 字段。真实 qwen3.7-plus 调用验证默认 effort=medium，Langfuse trace b24ef4b634847667317e87e240c82a6d 记录 policy_source=nexent_default，provider usage=30/196/226，finish_reason=stop。另修复未知 usage 子字段以 None 写入 OTEL 的告警。 |
|
补充修复后复测：最新 Langfuse trace 5bcc0a3ad7eada4f972aeeda93e86a77，effective reasoning=true/medium、policy_source=nexent_default；provider usage=30/200/230、reasoning=191、finish_reason=stop。未知 fresh-input/cache-write/visible-output 属性未写入 span，运行无 OTEL None 属性告警。 |
Codecov Report❌ Patch coverage is 📢 Thoughts on this report? Let us know! |
概要
对实际待发送的完整请求进行计量，并以 Provider 响应状态驱动安全恢复；输入只观测和触发计划动作，输出始终使用模型 profile 的最大输出值。
新增能力
lengthcontinuation。对旧行为的调整
input + max_tokens本地硬拒绝。max_tokens始终采用模型 profile 的max_output_tokens。对 Nexent 的提升
避免预算策略导致模型异常截断，同时通过 compaction、overflow 重建和安全 continuation 支撑 Agent 长时间运行。
规模与依赖
验证
git merge --squash；最终结果提交96cac2c15，测试通过且文件树与本分支一致。合并说明
这是堆叠变更的第三层；建议在前置 PR 合并后把 base 更新为
develop。