Skip to content

Preserve provider usage in request results - #4335

Open
Tanisha-Katara wants to merge 1 commit into
stanford-crfm:mainfrom
Tanisha-Katara:codex/preserve-provider-usage
Open

Tanisha-Katara wants to merge 1 commit into
stanford-crfm:mainfrom
Tanisha-Katara:codex/preserve-provider-usage

Conversation

@Tanisha-Katara

Copy link
Copy Markdown

Summary

Preserves provider-reported usage counters on RequestResult for OpenAI Chat/Completions/Embeddings, OpenAI Responses, and LiteLLM-backed calls.

The field is optional and omitted from serialized JSON when absent, so existing request results stay unchanged unless a provider actually returns usage.

Why

HELM currently estimates token costs from benchmark-side tokenization, but several providers now return more detailed usage counters. Those counters can include cache-specific token details such as cached input tokens and cache-write tokens. Keeping the raw provider usage attached to the result makes those details available to downstream analysis without asking HELM to interpret every provider's accounting semantics in this change.

For client paths that can make multiple provider calls for one HELM request, this keeps each provider usage dict under responses instead of silently dropping all but one.

Validation

  • uv run --python 3.12 pytest src/helm/common/test_codec.py -q
  • uv run --python 3.12 black --check src/helm/common/request.py src/helm/common/test_codec.py src/helm/clients/openai_client.py src/helm/clients/openai_responses_client.py src/helm/clients/litellm_client.py
  • uv run --python 3.12 python -m compileall -q src/helm/common/request.py src/helm/clients/openai_client.py src/helm/clients/openai_responses_client.py src/helm/clients/litellm_client.py
  • git diff --check

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant