feat(context): persist and present agent run usage - #3851
Conversation
There was a problem hiding this comment.
🟡 Changes recommended
There are blocking runtime issues (notably an ImportError risk in backend/agents/create_agent_info.py and a kwarg/signature mismatch in UnconfiguredContextRuntime) that can break production execution.
Once you've addressed the issues Copilot identified, you can request another Copilot review.
Pull request overview
This PR implements end-to-end observability for per-run context budget evidence and provider/turn usage: the SDK emits new content-free budget + usage events, the backend persists/aggregates them (including a new monitoring endpoint and DB column), and the frontend restores and displays them consistently for both SSE and historical sessions.
Changes:
- Add SDK-side usage aggregation (
turn_usage) and new observer event types (context_budget,llm_usage,turn_usage), plus optional emergency history archiving for overflow recovery. - Persist content-free request-budget evidence via a new migration and expose aggregated context-budget metrics via a monitoring endpoint.
- Extend frontend adapters and UI to parse, restore, and render the new metrics (including richer token usage panels and i18n additions).
File summaries
| File | Description |
|---|---|
sdk/nexent/core/models/usage_aggregation.py |
New deterministic aggregation for per-turn usage summaries. |
sdk/nexent/core/utils/observer.py |
Adds new ProcessType values and transformer wiring for budget/usage events. |
sdk/nexent/core/context_runtime/contracts.py |
Extends context runtime contracts/evidence for rebuild + archive/budget fields. |
sdk/nexent/core/agents/core_agent.py |
Adds context rebuild callbacks, emergency archive tool activation, and budget update emission. |
sdk/nexent/core/agents/nexent_agent.py |
Threads a per-turn ID and emits aggregated TURN_USAGE on final answer. |
sdk/nexent/core/agents/context/runtime.py |
Passes rebuild target/emergency archive flags through to context assembly. |
sdk/nexent/core/agents/context/manager.py |
Adds emergency archive working-set selection and request token metering integration. |
sdk/nexent/core/agents/context/archive.py |
Introduces run-local searchable archive + tool for emergency recall. |
sdk/nexent/core/agents/context/llm_summary.py |
Adds usage attribution kwargs for summarization calls. |
sdk/nexent/core/agents/context/long_term_memory_selector.py |
Adds usage attribution kwargs for long-term memory selection calls. |
sdk/nexent/core/agents/agent_model.py |
Extends ModelConfig with identity/capability/usage profile metadata. |
backend/agents/create_agent_info.py |
Resolves feature capabilities + provider usage profile; tightens safe-budget error mapping. |
backend/apps/monitoring_app.py |
Adds context-budget metrics query + /context-budget endpoint. |
backend/database/conversation_db.py |
Allows history summaries to be stored as structured markdown strings. |
backend/services/agent_service.py |
Deprecates legacy output/context overrides and normalizes update handling. |
deploy/sql/migrations/v2.6.0_0903_context_usage_observability.sql |
Adds context_budget_evidence JSONB column for persisted content-free evidence. |
frontend/types/chat.ts |
Adds ContextBudgetMetrics typing and wires it into TokenMetrics. |
frontend/const/chatConfig.ts |
Adds CONTEXT_BUDGET message type constant. |
frontend/lib/chatMessageExtractor.ts |
Parses/merges context budget events into step metrics and optionally surfaces them in step contents. |
frontend/components/common/tokenUsageIndicator.tsx |
Displays final request breakdown details when context_budget is present. |
frontend/app/[locale]/newchat/adapter/remote-chat-model-adapter.ts |
Parses/threads context_budget, llm_usage, and turn_usage through streaming runs. |
frontend/app/[locale]/newchat/adapter/conversation-thread-list-adapter.tsx |
Restores context_budget, llm_usage, and turn_usage for historical conversations. |
frontend/app/[locale]/newchat/ui/token-usage.tsx |
Major UI upgrade to show observed peak input + estimated composition + model call cost. |
frontend/app/[locale]/chat/streaming/chatStreamHandler.tsx |
Restores context_budget into reconstructed steps and surfaces timeline events. |
frontend/app/[locale]/chat/streaming/taskWindow.tsx |
Renders context budget optimization events in the task window stream. |
frontend/public/locales/zh/common.json |
Adds/updates i18n strings for token usage and context budget UX. |
frontend/public/locales/en/common.json |
Adds/updates i18n strings for token usage and context budget UX. |
test/sdk/core/models/test_usage_aggregation.py |
New tests for dedupe, partial field semantics, and context pressure. |
test/sdk/core/agents/test_core_agent.py |
New tests for context rebuild callbacks and emergency archive behavior. |
test/sdk/core/agents/test_context_manager_assembly.py |
Tests emergency archive working-set selection and recall tool injection. |
test/sdk/core/agents/test_context_helper_contracts.py |
Tests rebuild target tightening of budgets and failure reason mapping. |
test/sdk/core/agents/test_history_archive.py |
Tests archive redaction, unicode ranking, stable IDs, and kind validation. |
test/sdk/core/agents/test_nexent_agent.py |
Tests that model factory threading preserves identity metadata. |
test/backend/services/test_agent_service.py |
Tests atomic persistence ordering of usage units in stream batches. |
test/backend/database/test_conversation_db.py |
Tests structured markdown history summary parsing/persistence. |
test/backend/app/test_monitoring_app.py |
Tests new context budget metrics query logic. |
test/backend/agents/test_create_agent_info.py |
Extends stubs/tests for feature capability + budget resolver error mapping. |
test/deploy/test_context_budget_p3_migration.py |
Tests migration is idempotent/content-free and includes transaction boundaries. |
Review details
Suppressed comments (1)
sdk/nexent/core/context_runtime/contracts.py:197
UnconfiguredContextRuntime.prepare_final_answer()also lacks the newemergency_archivekwarg. This can surface as aTypeErrorduring rebuild attempts, masking the intended_UNCONFIGURED_RUNTIME_ERROR.
task: str,
final_answer_templates: Mapping[str, Mapping[str, str]],
tools: Sequence[ModelTool] | None = None,
target_input_budget_tokens: int | None = None,
) -> FinalContext:
- Files reviewed: 40/40 changed files
- Comments generated: 4
- Review effort level: Lite
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
| from nexent.core.models.prompt_cache import ( | ||
| resolve_prompt_cache_profile, | ||
| resolve_provider_usage_profile, | ||
| ) |
| def forward(self, query: str, top_k: int = 5, kinds: list[str] | None = None) -> dict[str, Any]: | ||
| return self.archive.search(query=query, top_k=top_k, kinds=kinds) |
| memory: AgentMemory, | ||
| current_run_start_idx: int, | ||
| tools: Sequence[ModelTool] | None = None, | ||
| target_input_budget_tokens: int | None = None, | ||
| ) -> FinalContext: |
| <div className="border-t border-gray-600 pt-1 mt-1 font-medium text-white"> | ||
| Final request breakdown | ||
| </div> | ||
| {Object.entries(budget.components) | ||
| .filter(([, value]) => value > 0) | ||
| .map(([name, value]) => ( | ||
| <div key={name} className="flex justify-between gap-4"> | ||
| <span className="text-gray-300"> | ||
| {name.replaceAll("_", " ")} | ||
| </span> | ||
| <span className="text-white">{formatNumber(value)}</span> | ||
| </div> | ||
| ))} | ||
| <div className="flex justify-between gap-4"> | ||
| <span className="text-gray-300">Count source</span> | ||
| <span className="text-white">{budget.count_source}</span> | ||
| </div> | ||
| {budget.compression.attempted && ( | ||
| <div className="flex justify-between gap-4"> | ||
| <span className="text-gray-300">Compaction saved</span> | ||
| <span className="text-green-300"> | ||
| {formatNumber(budget.compression.saved_tokens)} ( | ||
| {Math.round(budget.compression.ratio * 100)}%) | ||
| </span> | ||
| </div> | ||
| )} | ||
| {budget.recovery_state !== "not_needed" && ( | ||
| <div className="flex justify-between gap-4"> | ||
| <span className="text-gray-300">Recovery</span> | ||
| <span className="text-white">{budget.recovery_state}</span> | ||
| </div> |
29e5164 to
803ba1b
Compare
803ba1b to
713a8e1
Compare
713a8e1 to
c3e81fe
Compare
c3e81fe to
880af29
Compare
Codecov Report❌ Patch coverage is 📢 Thoughts on this report? Let us know! |
880af29 to
ecd7f4b
Compare
ecd7f4b to
0dc87c0
Compare
|
补充 2026-09-03 变更:AgentModel/NexentAgent 已预留 feature_preferences 透传契约,为后续前端关闭/调整并保存用户偏好提供稳定接入点;本 PR 不包含 UI 或数据库偏好持久化。四 PR 已重基到最新 origin/develop 41150e4,并完成本地四次 squash 演练;最终 tree 与本分支一致,相关独立测试合计 833 passed。 |
概要
将每次 Agent 运行的模型用量聚合、持久化并通过 SSE/历史会话恢复到统一前端详情面板,完成上下文治理的端到端可观测闭环。
新增能力
对旧行为的调整
对 Nexent 的提升
用户可以区分真实模型开销与估算的请求构成;运行时能够用观测锚点持续治理上下文,并在长会话压缩后保留可检索历史。
规模与依赖
验证
git merge --squash,最终结果提交6a8337293;测试/构建通过,累计树同时与本分支和 golden tip127f95500一致。已知测试约束
部分测试文件会在模块级替换
sys.modules,因此 OpenAI 与 Agent 核心套件需使用独立 pytest 进程运行;各自独立运行均通过。隔离 VM 的全量 runner 还会因未安装
unstructured/nest_asyncio、Multipass 挂载不支持临时目录内 Git/symlink,以及 sandbox 长测试超时而产生与本功能无关的失败;本系列涉及的失败均已逐个复现、修复并复验。合并说明
这是堆叠变更的最后一层。四个 PR 应按依赖顺序合并。