[Feat] Add OpenTelemetry tracing for RL rollout workflows#1946
Open
YanhuiDua wants to merge 7 commits into
Open
[Feat] Add OpenTelemetry tracing for RL rollout workflows#1946YanhuiDua wants to merge 7 commits into
YanhuiDua wants to merge 7 commits into
Conversation
This was referenced Jul 8, 2026
Collaborator
Author
|
@claude review |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
概述
这个 PR 为 XTuner RL rollout 增加 tracing 能力,并通过
xtuner.v1.rl.trace提供 XTuner 级别的 trace API。用户不需要直接依赖 OpenTelemetry SDK,也不需要关心 collector、exporter、Ray runtime env 等实现细节;只需要在 RL config 中开启 trace,并在自定义业务路径中使用 XTuner 提供的 trace API,即可把 controller、worker、
agent loop、judger、infer backend、session server 等路径上的执行信息串到同一条 trace 中。
这套能力主要用于排查 rollout 慢、失败、断链、异常传播和跨进程调用耗时问题。
使用方式
开启 trace
在 RL config 中配置
TraceConfig:Trace API
公共接口统一从 xtuner.v1.rl.trace 暴露。API 分为三层:basic API、XTuner endpoint API、context propagation API。
Basic API
Basic API 是 XTuner 封装的 trace primitive,适合普通本地阶段、临时埋点、自定义业务代码和手动 context 管理。
trace_span(name, attributes=None, parent_carrier=None)trace_function(name=None, attributes=None)trace_event(name, attributes=None)set_trace_attribute(key, value)set_trace_attributes(attributes)inject_trace_context(carrier=None)extract_trace_context(carrier)TraceContextattach_trace_context(context)TraceContext示例:
如果一段代码只是局部阶段,或者没有统一的 XTuner 业务生命周期,就应该直接使用 basic API,而不是新增的endpoint helper。
XTuner Endpoint API
Endpoint API 面向 XTuner 已知的稳定业务入口。它不只是创建 span,还会统一处理 target 绑定、initial/final attributes、status event、failure 记录和上游 context 接入等生命周期逻辑。
traced_rollout_endpoint(...)RolloutState为核心 target 的入口traced_agent_item_endpoint(...)traced_judger_endpoint(...)这类 API 适合框架内部的标准入口。普通业务代码通常不需要直接处理 rollout final attributes、agent item result attributes 或 judger result attributes。
Trace SKILL
Viewer使用方式
XTuner 提供面向 rollout 的 trace viewer。它不是新的 trace 后端,而是把 Jaeger 或本地 trace JSONL 中的原始 spans 按 rollout 语义聚合成样本视图。
sample状态总览

提供每个阶段的平均耗时、有哪些sample卡在了什么阶段等信息
每条sample调用链
