[REG-12] W0.7 - LLM span conventions - model, tokens, cost, latency

[singret]

Closes REG-12

Summary

Every LLM completion now produces a span using OTel's GenAI semantic conventions, so a bad answer, a cost spike, or a latency regression can all be traced back to the exact call that caused it.

What changed

• internal/integrations/llm/instrumented.go (new): instrumentedClient wraps any provider Client with a span per Complete() call. Uses semconv v1.34.0 specifically - the last version in this module's vendored set that still names the attribute gen_ai.system (later revisions rename it to gen_ai.provider.name; GenAI semconv is marked "Development" stability upstream and the names churn between versions - v1.34.0 is the one matching what REG-12 asks for).
• internal/integrations/llm/pricing.go (new): a small OSS per-model USD price table for the span's estimated cost. This is a separate computation from whatever Pro's CostTracker.RecordUsage charges internally (different repo, no shared code) - but both are fed the same Model/PromptTokens/CompletionTokens for a call, so the two agree numerically without needing to share an implementation.
• internal/integrations/llm/callmeta.go (new): CallMeta (agent name + incident ID) travels via context rather than as Complete() parameters, so tagging a call never requires changing the Client interface or touching all three provider implementations.
• internal/observability/genai.go (new): GenAIPromptRecordingEnabled(), gated on OTEL_INSTRUMENTATION_GENAI_CAPTURE_MESSAGE_CONTENT - the real upstream OTel GenAI instrumentation convention's own env var for this - mirroring SQLStatementRecordingEnabled's shape and rationale exactly.
• Wired into llm.New() for all three providers (openai, anthropic, ollama) and into all six ai_service.go call sites that reach a Client, each tagged with its own agent name ("Incident Summarizer", "Post-Mortem Drafter", ...) and incident ID where one is in scope.

Acceptance criteria

• Span per completion: gen_ai.system, gen_ai.request.model, gen_ai.usage.input_tokens, gen_ai.usage.output_tokens, gen_ai.operation.name
• prompt_hash (sha256 of rendered system+user messages) as a span attribute
• Estimated cost in USD as a span attribute, from a per-model price table
• incident.id and gen_ai.agent.name where applicable (the real GenAI semconv key for "named agent/persona", used instead of inventing a custom agent.name)
• Applies to all three providers, not just OpenAI
• Prompt/completion text NOT recorded by default; opt-in via OTEL_INSTRUMENTATION_GENAI_CAPTURE_MESSAGE_CONTENT=true for local debugging only
• Already wired into CostTracker.RecordUsage at the handler level (pre-existing, using the same Model/Usage values this PR's spans use) - verified unaffected

Found and flagged, not fixed here

buildAnswerQuestionPrompt dereferences current *models.Incident unconditionally even though AnswerQuestion's own signature documents it as optional (nil-able). This panics if ever called with current == nil. It's dormant today - the sole caller in slack_event_handler.go already guards incident == nil before calling AnswerQuestion - but is a real defensive gap for the next caller. Out of scope for this ticket (span conventions, not this bug); flagging rather than silently fixing or silently leaving undocumented.

Verification

• go build ./..., go vet ./..., gofmt -l clean on all touched files
• Full test suite green, plus -shuffle=on on all touched packages
• golangci-lint run ./...: 0 issues
• New code at 100% coverage (internal/integrations/llm package: 90.8% overall)
• Live-verified against a real Jaeger collector, through the real llm.New() path, for all three providers: correct system/model/token counts/cost/agent.name/incident.id/prompt_hash tags, zero span events (no PII) by default

[mintlify]

Preview deployment for your docs. Learn more about Mintlify Previews.

Project	Status	Preview	Updated (UTC)
fluidify	🟢 Ready	View Preview	Sep 1, 2026, 9:54 AM

💡 Tip: Enable Workflows to automatically generate PRs for you.