Skip to content

feat(observability): 接入 Prometheus 指标与 OpenTelemetry/Tempo 链路 - #360

Merged
gac0812 merged 12 commits into
1024XEngineer:mainfrom
gac0812:feat/observability-prometheus-tempo
Aug 27, 2026
Merged

gac0812 merged 12 commits into
1024XEngineer:mainfrom
gac0812:feat/observability-prometheus-tempo

Conversation

@gac0812

@gac0812 gac0812 commented Aug 24, 2026

Copy link
Copy Markdown
Contributor

Summary

  • 后端接入 Prometheus 指标(QPS、延迟、连接数、工具调用、外部依赖、数据库)和 OpenTelemetry → Tempo 链路(每轮对话、工具、ASR/LLM/TTS 耗时)。
  • Grafana 只是看数据的界面,不是这次交付目标;代码改 HTTP / WebSocket / Agent / 外部 SDK,是为了打点和打 span,不改业务功能。
  • GET /metrics 供 Prometheus scrape;配置 TIMEFLOW_OTEL_EXPORTER_OTLP_ENDPOINT 后才导出 trace,未配置则 tracing 关闭。
  • 指标 label 与 span attribute 使用封闭枚举,不写 transcript、session id、坐标或凭证。

Closes #358

Test plan

  • GET /metrics 返回 Prometheus text,且不把 /metrics 自己算进业务 HTTP QPS
  • HTTP / WebSocket / 语音阶段 / 工具 / 外部依赖 / 数据库连接池都有对应指标
  • 一轮对话结束后,Tempo 能立刻看到 voice.turn 以及嵌套的 tool / ASR / LLM / TTS / DB span,不必等 WebSocket 断开
  • 本地 Grafana 能画出折线图和 Trace 表(效果图见 feat(observability): 接入 Prometheus 指标与 OpenTelemetry/Tempo 链路 #358)
  • 业务行为不变:鉴权、快照、提醒、语音对话语义未改

Verification

cd backend
bash scripts/check.sh

本地结果:1090 passed, 49 skipped,coverage 96.73%(门槛 95%)。

覆盖要点:

  • HTTP 中间件按路由模板计数;/metrics 不计入业务 QPS
  • voice.turn 单独成 trace,工具 span 保持调用顺序
  • 未知工具名折叠为 other;失败 turn 标记 ERROR
  • settings 在本地 .env 有 Tempo 地址时仍能断言 OTLP 开关

效果对照

Grafana 把 Prometheus / Tempo 画出来即可,截图不进源码树:

Overview + HTTP

Overview and HTTP

WebSocket + Voice / tools

WebSocket and voice

External / database

External and database

Traces(Tempo)

Tempo trace tables

单轮 voice.turn 瀑布图

Tempo voice.turn waterfall

HTTP/WebSocket/Agent/外部 SDK 只打点和打 span,方便定位一轮对话卡在 ASR、LLM 还是 TTS;Grafana 只负责把数据画出来。
Grafana 需要当前卡在 ASR/LLM/TTS 的人数,以及 UI 挂断、超时、道别的区分;session_id 只留进程内,不进 Prometheus label。
@gac0812

gac0812 commented Aug 25, 2026

Copy link
Copy Markdown
Contributor Author

已补上服务端会话数据埋点(占用阶段 Gauge + 挂断原因 ui_hangup / idle_timeout / tool_end)。session_id 只留进程内,不进 Prometheus label。

打断行为拆到后续草稿 PR,不放在本 PR。

Grafana 按登录页账号筛 turn/tool 轨迹;username 只作为 span 属性,不进 Prometheus label。
@gac0812
gac0812 marked this pull request as ready for review August 26, 2026 08:01
@codecov

codecov Bot commented Aug 26, 2026 •

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 95.87799% with 50 lines in your changes missing coverage. Please review.

Files with missing lines Patch % Lines
...w/infrastructure/external/llm/openai_compatible.py 82.14% 9 Missing and 1 partial ⚠️
...ackend/src/timeflow/intelligence/composed/agent.py 80.00% 5 Missing and 3 partials ⚠️
...flow/infrastructure/external/tts/qwen_audio_tts.py 75.00% 6 Missing ⚠️
.../timeflow/infrastructure/observability/database.py 93.10% 2 Missing and 4 partials ⚠️
...w/infrastructure/external/location/tencent_maps.py 86.95% 2 Missing and 1 partial ⚠️
...low/infrastructure/external/realtime/qwen_audio.py 91.42% 2 Missing and 1 partial ⚠️
...ackend/src/timeflow/intelligence/realtime/agent.py 86.36% 3 Missing ⚠️
...c/timeflow/intelligence/realtime/schedule_tools.py 72.72% 3 Missing ⚠️
backend/src/timeflow/gateway/websocket/endpoint.py 96.61% 1 Missing and 1 partial ⚠️
...eflow/infrastructure/external/asr/qwen_realtime.py 91.66% 2 Missing ⚠️
... and 3 more
Flag Coverage Δ
backend 96.57% <95.87%> (-0.17%) ⬇️
frontend 89.80% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

Files with missing lines Coverage Δ
backend/src/timeflow/composition.py 97.14% <100.00%> (+0.08%) ⬆️
backend/src/timeflow/data/database.py 100.00% <100.00%> (ø)
backend/src/timeflow/data/repositories/account.py 95.34% <100.00%> (+0.34%) ⬆️
backend/src/timeflow/gateway/http/auth.py 95.86% <100.00%> (+0.29%) ⬆️
...ackend/src/timeflow/gateway/http/reminder_state.py 100.00% <100.00%> (ø)
...end/src/timeflow/gateway/http/schedule_snapshot.py 100.00% <100.00%> (ø)
...end/src/timeflow/gateway/observability/__init__.py 100.00% <100.00%> (ø)
backend/src/timeflow/gateway/observability/http.py 100.00% <100.00%> (ø)
...end/src/timeflow/gateway/observability/sessions.py 100.00% <100.00%> (ø)
...nd/src/timeflow/gateway/observability/websocket.py 100.00% <100.00%> (ø)
... and 25 more
🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

此前整棵 /observability/ 被 ignore,登录账号筛选只留在本机 Grafana。把看板 JSON 纳入仓库,Tempo 表选出 timeflow.username。

@fennoai fennoai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed the complete observability diff, including HTTP/WebSocket lifecycle, agent/provider instrumentation, database hooks, settings, and the new tests. The implementation is generally cohesive, but the username enrichment path can make otherwise successful voice turns fail when the database is temporarily unavailable. python3 -m compileall -q backend/src backend/tests and git diff --check pass; backend/scripts/check.sh could not run because uv is not installed in the review environment.

Comment thread backend/src/timeflow/main.py Outdated
合入 main 后该行超长,CI 的 ruff format --check 失败。
mypy 将 Protocol 标注视为对外类型,导致 composition root 无法调用 bind_username_lookup。
覆盖 composition root 的账户查找、NoOp 占用接口,以及握手/播放/数据库监听的边界分支,把 patch 覆盖率抬过 Codecov 门槛。
数据库短暂不可用时记录脱敏诊断,span 使用 unknown 且不写入缓存,恢复后下一轮仍可补上登录名。
配置改为容器路径和 Docker 服务名,去掉本机绝对路径,便于和 API 一起上云。
@gac0812
gac0812 merged commit 07cec6e into 1024XEngineer:main Aug 27, 2026
5 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

feat(observability): 接入 Prometheus 指标与 OpenTelemetry/Tempo 链路

2 participants