feat: record time-to-first-token on streaming generations#29
Merged
Conversation
Generation.record_first_token() adds a gen_ai.first_token span event at the moment the first streamed chunk arrives; the backend derives TTFT as event time minus span start (the spans table already carries span events — no schema change). Idempotent, so streaming loops can call it unconditionally per chunk; a no-op on ended or non-recording spans. Event name follows the gen_ai.* style (no client-side semconv exists — OTel standardizes only the server-side time_to_first_token metric); design precedent is Langfuse's completion_start_time. Auto-instrumented traffic: the OpenInference OpenAI instrumentor already emits its own "First Token Stream Event" on streaming spans — TTFT derivation must coalesce both names (backend-normalization surface, same bucket as GLA2-133). The anthropic/langchain/litellm instrumentors emit no first-token marker at all.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Implements GLA2-175.
What
Generation.record_first_token()— call it from a streaming loop when the first content chunk arrives:It adds a
gen_ai.first_tokenspan event (constant insemconv.py). Idempotent (only the first call records), no-op on ended/non-recording spans.Why an event, and why this name
spanstable already stores span events, so TTFT is derivable today asevent timestamp − span Timestamp.gen_ai.server.time_to_first_tokenmetric, emitted by model servers); the event name follows thegen_ai.*style per our convention-native rule. Design precedent: Langfuse'scompletion_start_time.Auto-instrumented (OpenInference) traffic — deliberately NOT normalized here
Surveyed all five registry instrumentors:
"First Token Stream Event"on the first streamed chunk (_stream.py).So TTFT derivation must coalesce both event names. That normalization belongs backend-side (same bucket as GLA2-133's
llm.token_count.*gap): collector users and other SDKs bypass our exporter anyway, so an SDK-side rename would duplicate work the ingest layer needs regardless. Reference derivation:Testing
TDD: 4 new tests (event lands between span start/end, idempotency, no-op after
end(), manual lifecycle) written first and watched fail. Full suite 121 passed; ruff + format + mypy --strict clean.