Skip to content

feat: record time-to-first-token on streaming generations#29

Merged
PabloPardoGarcia merged 1 commit into
mainfrom
pablo/gla2-175-record-first-token
Jul 15, 2026
Merged

feat: record time-to-first-token on streaming generations#29
PabloPardoGarcia merged 1 commit into
mainfrom
pablo/gla2-175-record-first-token

Conversation

@PabloPardoGarcia

Copy link
Copy Markdown
Member

Implements GLA2-175.

What

Generation.record_first_token() — call it from a streaming loop when the first content chunk arrives:

gen = glassflow.start_generation("chat", model="gpt-4o")
stream = client.chat.completions.create(..., stream=True)
for chunk in stream:
    gen.record_first_token()   # idempotent — safe to call per chunk
    ...
gen.end()

It adds a gen_ai.first_token span event (constant in semconv.py). Idempotent (only the first call records), no-op on ended/non-recording spans.

Why an event, and why this name

  • The backend needs no schema change: the spans table already stores span events, so TTFT is derivable today as event timestamp − span Timestamp.
  • No client-side semconv exists for this (OTel standardizes only the server-side gen_ai.server.time_to_first_token metric, emitted by model servers); the event name follows the gen_ai.* style per our convention-native rule. Design precedent: Langfuse's completion_start_time.

Auto-instrumented (OpenInference) traffic — deliberately NOT normalized here

Surveyed all five registry instrumentors:

  • openai: already emits its own "First Token Stream Event" on the first streamed chunk (_stream.py).
  • anthropic / langchain / litellm (and llama-index): no first-token marker at all — their streaming traces have no TTFT until upstream adds it or we wrap provider streams (the ticket's follow-up).

So TTFT derivation must coalesce both event names. That normalization belongs backend-side (same bucket as GLA2-133's llm.token_count.* gap): collector users and other SDKs bypass our exporter anyway, so an SDK-side rename would duplicate work the ingest layer needs regardless. Reference derivation:

SELECT GenAiRequestModel,
       quantile(0.95)((
         toUnixTimestamp64Nano(Events.Timestamp[arrayFirstIndex(n -> n IN ('gen_ai.first_token', 'First Token Stream Event'), Events.Name)])
         - toUnixTimestamp64Nano(Timestamp)) / 1e6) AS ttft_p95_ms
FROM spans
WHERE arrayExists(n -> n IN ('gen_ai.first_token', 'First Token Stream Event'), Events.Name)
GROUP BY GenAiRequestModel

Testing

TDD: 4 new tests (event lands between span start/end, idempotency, no-op after end(), manual lifecycle) written first and watched fail. Full suite 121 passed; ruff + format + mypy --strict clean.

Generation.record_first_token() adds a gen_ai.first_token span event at
the moment the first streamed chunk arrives; the backend derives TTFT
as event time minus span start (the spans table already carries span
events — no schema change). Idempotent, so streaming loops can call it
unconditionally per chunk; a no-op on ended or non-recording spans.

Event name follows the gen_ai.* style (no client-side semconv exists —
OTel standardizes only the server-side time_to_first_token metric);
design precedent is Langfuse's completion_start_time.

Auto-instrumented traffic: the OpenInference OpenAI instrumentor
already emits its own "First Token Stream Event" on streaming spans —
TTFT derivation must coalesce both names (backend-normalization
surface, same bucket as GLA2-133). The anthropic/langchain/litellm
instrumentors emit no first-token marker at all.
@PabloPardoGarcia PabloPardoGarcia merged commit 6c1ecaa into main Jul 15, 2026
7 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant