Skip to content

feat(context): persist and present agent run usage - #3851

Open
JasonW404 wants to merge 3 commits into
split/context-request-recoveryfrom
split/context-usage-observability
Open

feat(context): persist and present agent run usage#3851
JasonW404 wants to merge 3 commits into
split/context-request-recoveryfrom
split/context-usage-observability

Conversation

@JasonW404

@JasonW404 JasonW404 commented Sep 3, 2026

Copy link
Copy Markdown
Member

概要

将每次 Agent 运行的模型用量聚合、持久化并通过 SSE/历史会话恢复到统一前端详情面板,完成上下文治理的端到端可观测闭环。

新增能力

  • 聚合最终成功 completion 的模型调用开销,同时保留运行级数据质量和物理调用计数。
  • 持久化 usage,历史会话与实时会话使用同一语义适配器。
  • 展示 API 实测峰值输入、显式未使用空间,以及独立标注的估算上下文构成。
  • 横向多列详情面板、响应式定位、键盘/外部关闭和中英文国际化。
  • 增加历史归档搜索,使 compaction 后的信息仍可恢复。

对旧行为的调整

  • 标题统一为“本轮 Agent 运行使用详情”。
  • “模型调用开销”只展示 API 返回字段;缺失值显示为破折号,不再用本地估算填充。
  • 删除 Provider overhead 和逐次 Provider 调用折叠明细。
  • 中途错误不再清空最终成功响应的 usage;结束原因和计数来源按最终响应语义展示。

对 Nexent 的提升

用户可以区分真实模型开销与估算的请求构成;运行时能够用观测锚点持续治理上下文,并在长会话压缩后保留可检索历史。

规模与依赖

验证

  • 后端 create-agent 236、monitoring API 15、conversation DB 136 个测试通过。
  • ContextManager 38、Agent core 159、Nexent Agent 240、历史归档及聚合专项测试通过。
  • OpenAI 适配器 91 个回归测试及 Agent service 定向测试通过。
  • TypeScript type-check 与 Next.js 15.5.25 生产构建通过。
  • 本地完成第四层 git merge --squash,最终结果提交 6a8337293;测试/构建通过,累计树同时与本分支和 golden tip 127f95500 一致。
  • 合并默认 main/sub model 的重复构造逻辑,消除 Sonar 新代码重复率门禁。
  • 全量 runner 审计后补充通过 automation 38、旧 ContextManager 调用 2 个兼容测试;Agent service 464 个定向测试通过。

已知测试约束

部分测试文件会在模块级替换 sys.modules,因此 OpenAI 与 Agent 核心套件需使用独立 pytest 进程运行;各自独立运行均通过。

隔离 VM 的全量 runner 还会因未安装 unstructured/nest_asyncio、Multipass 挂载不支持临时目录内 Git/symlink,以及 sandbox 长测试超时而产生与本功能无关的失败;本系列涉及的失败均已逐个复现、修复并复验。

合并说明

这是堆叠变更的最后一层。四个 PR 应按依赖顺序合并。

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

There are blocking runtime issues (notably an ImportError risk in backend/agents/create_agent_info.py and a kwarg/signature mismatch in UnconfiguredContextRuntime) that can break production execution.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Pull request overview

This PR implements end-to-end observability for per-run context budget evidence and provider/turn usage: the SDK emits new content-free budget + usage events, the backend persists/aggregates them (including a new monitoring endpoint and DB column), and the frontend restores and displays them consistently for both SSE and historical sessions.

Changes:

  • Add SDK-side usage aggregation (turn_usage) and new observer event types (context_budget, llm_usage, turn_usage), plus optional emergency history archiving for overflow recovery.
  • Persist content-free request-budget evidence via a new migration and expose aggregated context-budget metrics via a monitoring endpoint.
  • Extend frontend adapters and UI to parse, restore, and render the new metrics (including richer token usage panels and i18n additions).
File summaries
File Description
sdk/nexent/core/models/usage_aggregation.py New deterministic aggregation for per-turn usage summaries.
sdk/nexent/core/utils/observer.py Adds new ProcessType values and transformer wiring for budget/usage events.
sdk/nexent/core/context_runtime/contracts.py Extends context runtime contracts/evidence for rebuild + archive/budget fields.
sdk/nexent/core/agents/core_agent.py Adds context rebuild callbacks, emergency archive tool activation, and budget update emission.
sdk/nexent/core/agents/nexent_agent.py Threads a per-turn ID and emits aggregated TURN_USAGE on final answer.
sdk/nexent/core/agents/context/runtime.py Passes rebuild target/emergency archive flags through to context assembly.
sdk/nexent/core/agents/context/manager.py Adds emergency archive working-set selection and request token metering integration.
sdk/nexent/core/agents/context/archive.py Introduces run-local searchable archive + tool for emergency recall.
sdk/nexent/core/agents/context/llm_summary.py Adds usage attribution kwargs for summarization calls.
sdk/nexent/core/agents/context/long_term_memory_selector.py Adds usage attribution kwargs for long-term memory selection calls.
sdk/nexent/core/agents/agent_model.py Extends ModelConfig with identity/capability/usage profile metadata.
backend/agents/create_agent_info.py Resolves feature capabilities + provider usage profile; tightens safe-budget error mapping.
backend/apps/monitoring_app.py Adds context-budget metrics query + /context-budget endpoint.
backend/database/conversation_db.py Allows history summaries to be stored as structured markdown strings.
backend/services/agent_service.py Deprecates legacy output/context overrides and normalizes update handling.
deploy/sql/migrations/v2.6.0_0903_context_usage_observability.sql Adds context_budget_evidence JSONB column for persisted content-free evidence.
frontend/types/chat.ts Adds ContextBudgetMetrics typing and wires it into TokenMetrics.
frontend/const/chatConfig.ts Adds CONTEXT_BUDGET message type constant.
frontend/lib/chatMessageExtractor.ts Parses/merges context budget events into step metrics and optionally surfaces them in step contents.
frontend/components/common/tokenUsageIndicator.tsx Displays final request breakdown details when context_budget is present.
frontend/app/[locale]/newchat/adapter/remote-chat-model-adapter.ts Parses/threads context_budget, llm_usage, and turn_usage through streaming runs.
frontend/app/[locale]/newchat/adapter/conversation-thread-list-adapter.tsx Restores context_budget, llm_usage, and turn_usage for historical conversations.
frontend/app/[locale]/newchat/ui/token-usage.tsx Major UI upgrade to show observed peak input + estimated composition + model call cost.
frontend/app/[locale]/chat/streaming/chatStreamHandler.tsx Restores context_budget into reconstructed steps and surfaces timeline events.
frontend/app/[locale]/chat/streaming/taskWindow.tsx Renders context budget optimization events in the task window stream.
frontend/public/locales/zh/common.json Adds/updates i18n strings for token usage and context budget UX.
frontend/public/locales/en/common.json Adds/updates i18n strings for token usage and context budget UX.
test/sdk/core/models/test_usage_aggregation.py New tests for dedupe, partial field semantics, and context pressure.
test/sdk/core/agents/test_core_agent.py New tests for context rebuild callbacks and emergency archive behavior.
test/sdk/core/agents/test_context_manager_assembly.py Tests emergency archive working-set selection and recall tool injection.
test/sdk/core/agents/test_context_helper_contracts.py Tests rebuild target tightening of budgets and failure reason mapping.
test/sdk/core/agents/test_history_archive.py Tests archive redaction, unicode ranking, stable IDs, and kind validation.
test/sdk/core/agents/test_nexent_agent.py Tests that model factory threading preserves identity metadata.
test/backend/services/test_agent_service.py Tests atomic persistence ordering of usage units in stream batches.
test/backend/database/test_conversation_db.py Tests structured markdown history summary parsing/persistence.
test/backend/app/test_monitoring_app.py Tests new context budget metrics query logic.
test/backend/agents/test_create_agent_info.py Extends stubs/tests for feature capability + budget resolver error mapping.
test/deploy/test_context_budget_p3_migration.py Tests migration is idempotent/content-free and includes transaction boundaries.
Review details

Suppressed comments (1)

sdk/nexent/core/context_runtime/contracts.py:197

  • UnconfiguredContextRuntime.prepare_final_answer() also lacks the new emergency_archive kwarg. This can surface as a TypeError during rebuild attempts, masking the intended _UNCONFIGURED_RUNTIME_ERROR.
        task: str,
        final_answer_templates: Mapping[str, Mapping[str, str]],
        tools: Sequence[ModelTool] | None = None,
        target_input_budget_tokens: int | None = None,
    ) -> FinalContext:
  • Files reviewed: 40/40 changed files
  • Comments generated: 4
  • Review effort level: Lite

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Comment thread backend/agents/create_agent_info.py Outdated
Comment on lines +20 to +23
from nexent.core.models.prompt_cache import (
resolve_prompt_cache_profile,
resolve_provider_usage_profile,
)
Comment on lines +146 to +147
def forward(self, query: str, top_k: int = 5, kinds: list[str] | None = None) -> dict[str, Any]:
return self.archive.search(query=query, top_k=top_k, kinds=kinds)
Comment on lines 180 to 184
memory: AgentMemory,
current_run_start_idx: int,
tools: Sequence[ModelTool] | None = None,
target_input_budget_tokens: int | None = None,
) -> FinalContext:
Comment on lines +91 to +121
<div className="border-t border-gray-600 pt-1 mt-1 font-medium text-white">
Final request breakdown
</div>
{Object.entries(budget.components)
.filter(([, value]) => value > 0)
.map(([name, value]) => (
<div key={name} className="flex justify-between gap-4">
<span className="text-gray-300">
{name.replaceAll("_", " ")}
</span>
<span className="text-white">{formatNumber(value)}</span>
</div>
))}
<div className="flex justify-between gap-4">
<span className="text-gray-300">Count source</span>
<span className="text-white">{budget.count_source}</span>
</div>
{budget.compression.attempted && (
<div className="flex justify-between gap-4">
<span className="text-gray-300">Compaction saved</span>
<span className="text-green-300">
{formatNumber(budget.compression.saved_tokens)} (
{Math.round(budget.compression.ratio * 100)}%)
</span>
</div>
)}
{budget.recovery_state !== "not_needed" && (
<div className="flex justify-between gap-4">
<span className="text-gray-300">Recovery</span>
<span className="text-white">{budget.recovery_state}</span>
</div>
@JasonW404
JasonW404 force-pushed the split/context-usage-observability branch from 29e5164 to 803ba1b Compare September 3, 2026 06:46
@JasonW404
JasonW404 force-pushed the split/context-usage-observability branch from 803ba1b to 713a8e1 Compare September 3, 2026 07:32
@JasonW404
JasonW404 force-pushed the split/context-usage-observability branch from 713a8e1 to c3e81fe Compare September 3, 2026 08:43
@JasonW404
JasonW404 force-pushed the split/context-usage-observability branch from c3e81fe to 880af29 Compare September 3, 2026 08:53
@JasonW404
JasonW404 force-pushed the split/context-usage-observability branch from 880af29 to ecd7f4b Compare September 3, 2026 09:09
@JasonW404
JasonW404 force-pushed the split/context-usage-observability branch from ecd7f4b to 0dc87c0 Compare September 3, 2026 09:11
@JasonW404

Copy link
Copy Markdown
Member Author

补充 2026-09-03 变更:AgentModel/NexentAgent 已预留 feature_preferences 透传契约,为后续前端关闭/调整并保存用户偏好提供稳定接入点;本 PR 不包含 UI 或数据库偏好持久化。四 PR 已重基到最新 origin/develop 41150e4,并完成本地四次 squash 演练;最终 tree 与本分支一致,相关独立测试合计 833 passed。

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants