Skip to content

feat(intelligence): add Qwen TTS streaming pipeline - #205

Merged
Wintercom merged 6 commits into
1024XEngineer:mainfrom
yyy-router:feature/intelligence-streaming-pipeline
Aug 12, 2026
Merged

Wintercom merged 6 commits into
1024XEngineer:mainfrom
yyy-router:feature/intelligence-streaming-pipeline

Conversation

@yyy-router

@yyy-router yyy-router commented Aug 12, 2026 •

Copy link
Copy Markdown
Contributor

功能概述

  • 增加供应商无关的 TTS 公共契约与环境配置
  • 使用阿里云官方原始 WebSocket 协议实现 Qwen-Audio-TTS Provider
  • 增加增量文本句段切分,在完整句段形成后立即送入 TTS
  • 将 Agent 已确认可播报的回复流式转换为 PCM 音频
  • 保留 Agent 的安全边界,工具调用轮次文本和未经业务验证的结果不会进入 TTS
  • 使用 Fake Provider 覆盖协议顺序、取消、错误脱敏、文本切分和 Agent→TTS 流式时序

协议与实现边界

本 PR 严格使用官方文档中定义的协议元素:

  • run-task
  • continue-task
  • finish-task
  • task-started
  • result-generated
  • task-finished
  • task-failed
  • WebSocket Binary 音频帧

第一版采用以下任务模型:

一个可播报轮次
→ 一个 task_id
→ run-task
→ task-started
→ continue-task(segment_0)
→ continue-task(segment_1)
→ ...
→ finish-task
→ task-finished

具体约束:

  • 同一语音轮次中的所有客户端事件复用一个 task_id
  • 多个文本句段按顺序发送 continue-task
  • 输出 PCM 24kHz、单声道、16bit 音频
  • Provider 在后续文本仍在到达时即可返回首个音频块
  • 音频按 Provider 返回顺序向上游输出
  • 不引入句段级独立 task、并发合成、乱序重排或跨任务连接池

Agent 可播报边界

允许进入 TTS:

  • 最终无工具调用轮次的 AgentTextDelta
  • AgentQuestion.speech_text

禁止进入 TTS:

  • System Prompt
  • ASR 转写
  • Tool Call、Tool Result 和工具参数
  • AgentCompleted
  • 工具调用轮次中的中间文本
  • 原始异常、API Key、请求头和数据库错误详情

本 PR 不包含

  • 完整 ASR → Agent → TTS 产品集成
  • ComposedVoiceAgent 会话生命周期编排
  • Realtime Voice Agent
  • Gateway Composition Root 切换
  • 客户端用户打断及 voice.tts.cancel 协议
  • 地图 Provider
  • 完整提醒播报链路
  • 长连接池或跨 task dispatcher

验证结果

  • uv run ruff check .:通过
  • uv run ruff format --check .:通过
  • uv run mypy:通过
  • TTS、切分器、Pipeline、Settings 和架构定向测试:通过
  • 完整 pytest 功能断言:通过(22 项跳过)
  • Alembic:唯一 head 为 20260810_0005
  • uv lock --check:通过

真实 Qwen TTS 冒烟

单句场景:

  • 首音频延迟:529.9 ms
  • 音频块:7
  • PCM 字节:136,800

同一 task 两个句段:

  • 首音频延迟:413.5 ms
  • 音频块:12
  • PCM 字节:217,440

最终结果:SMOKE_RESULT=passed

已知基线问题

仓库当前配置的全局覆盖率门槛为 95%。本分支完整测试覆盖率为 92.26%,同步后的 upstream/main 在尚未加入 TTS 代码时基线为 91.89%。本 PR 提高了总体覆盖率,没有降低门槛,也没有混入其他主线模块的无关补测。

Part of #183

@vercel

vercel Bot commented Aug 12, 2026 •

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated (UTC)
timeflow Ready Ready Preview Aug 12, 2026 6:57am

@yyy-router
yyy-router marked this pull request as ready for review August 12, 2026 06:35

@fennoai fennoai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Found one high-confidence reliability issue in the new speech pipeline. Focused tests were not runnable because uv and pytest are unavailable in the environment; python3 -m compileall -q backend/src backend/tests passed.

Comment thread backend/src/timeflow/intelligence/speech/pipeline.py Outdated
Comment thread backend/src/timeflow/infrastructure/external/tts/qwen_audio_tts.py
@Wintercom
Wintercom merged commit 071d07d into 1024XEngineer:main Aug 12, 2026
5 checks passed
gac0812 pushed a commit to gac0812/Tfgxs that referenced this pull request Aug 13, 2026
Bring in 1024XEngineer#198/1024XEngineer#199/1024XEngineer#205/1024XEngineer#210/1024XEngineer#213/1024XEngineer#217 while keeping the local native
reminder stack, strength delivery, and alarm event hydration.

This branch was successfully deployed

1 active deployment
Preview — 81e96deb Deployed Aug 12, 2026 by vercel[bot]
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants