Skip to content

[bot] Anthropic usage extractor reads non-existent reasoning_tokens field instead of output_tokens_details.thinking_tokens #98

Description

@braintrust-bot

<!-- provider-gap-audit: anthropic-thinking-tokens-field-mismatch -->

Summary

extract_anthropic_usage() in src/extractors.rs looks for a field literally named reasoning_tokens — both as a flat usage.reasoning_tokens field and nested under output_tokens_details.reasoning_tokens — to populate reasoning/thinking token counts for Anthropic extended-thinking responses. Neither field exists in Anthropic's real API. The actual field, per Anthropic's own documentation, is usage.output_tokens_details.thinking_tokens. As a result, completion_reasoning_tokens is always None for genuine Anthropic extended-thinking API responses — this is not a partial-detail gap, it's a complete miss caused by a wrong field name.

What is wrong

src/extractors.rs lines 135-138 read a flat top-level usage.reasoning_tokens field that Anthropic never emits.

src/extractors.rs lines 140-156 build completion_tokens_details from usage.get("output_tokens_details"), and at line 150 read details.get("reasoning_tokens") — again the wrong key. The correct key per Anthropic's docs is thinking_tokens.

This mistaken assumption is also baked into the test suite: the extracts_anthropic_cache_metrics test (src/extractors.rs, around line 299) asserts a "reasoning_tokens": 12 fixture value and expects completion_reasoning_tokens() == Some(12) — encoding the incorrect field name as expected behavior rather than testing against Anthropic's real response shape.

Braintrust docs status

unclear — checked https://www.braintrust.dev/docs/integrations/ai-providers/anthropic (2026-09-02). The page lists thinking as a captured request parameter (Python/Go/Ruby sections), but none of the documented usage metrics tables include a thinking_tokens or reasoning_tokens field for any language SDK. Braintrust does not explicitly state whether extended-thinking output token counts are captured as a metric.

Upstream sources

Local files inspected

  • src/extractors.rs lines 92-188 (extract_anthropic_usage), specifically lines 135-138 and 140-156
  • src/extractors.rs around line 299 (extracts_anthropic_cache_metrics test fixture asserting the incorrect field name)
  • src/types.rs (CompletionTokensDetails, UsageMetrics structs)
  • Full codebase grep for thinking_tokens — zero results

Fix sketch

Read thinking_tokens as the primary (or aliased) key when parsing Anthropic's output_tokens_details, e.g. details.get("thinking_tokens").or_else(|| details.get("reasoning_tokens")).

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions