-
Notifications
You must be signed in to change notification settings - Fork 37
docs(analytics): server-tool accounting, 367-day cap, region + query behavior #94
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: main
Are you sure you want to change the base?
Changes from all commits
e7acb5d
483d0c9
9d6892d
ac1cead
1ecb3eb
c8a3628
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -122,13 +122,16 @@ The query endpoint returns an array of data rows. Each row is a flat object with | |
|
|
||
| When interpreting results for the user: | ||
| - **Spend metrics** (`total_usage`, `credits_usage`, `openrouter_usage`, `byok_usage`, `byok_fees`, `usage_upstream`, `usage_cache`, `usage_web`, `usage_upstream_web`, `usage_file`, `usage_upstream_file`, `usage_web_fetch`, `usage_upstream_web_fetch`) are in USD. `total_usage` includes BYOK inference cost. `usage_data` is typically negative (a data logging discount) | ||
| - **Blended cost** (`blended_cost_per_million_tokens`) is the blended inference cost per 1M tokens, including applicable BYOK inference usage | ||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. 🔍 blended_cost_per_million_tokens described only in two of three skills The metric is added to the interpretation list here and to the numeric-types note in Was this helpful? React with 👍 or 👎 to provide feedback.
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. |
||
| - **Token counts** (`tokens_total`, `tokens_prompt`, `tokens_completion`) are in native model tokens | ||
| - **Latency** (`avg_latency`, `p50_latency`, etc.) is in milliseconds | ||
| - **Rates** (`cache_hit_rate`) are 0–1 ratios | ||
| - **Throughput** (`avg_throughput`) is tokens per second | ||
| - When `granularity` is set, rows include a `date__<granularity>` field for the time bucket (e.g., `date__day`, `date__hour`, `date__month`) | ||
| - When `granularity` is set, rows include a source-dependent time-bucket field: `date__<granularity>` for MV-backed queries or `created_at__<granularity>` for raw generations/classification queries (detect either prefix) | ||
| - **Label resolution**: dimensions `api_key_id`, `app`, `user`, and `workspace` have their raw IDs replaced with human-readable names (key name, app title, user name, workspace name) directly in the data rows; `generation_id` and `session_id` return raw values | ||
| - **Label fallbacks**: `app = -1` is `Unknown`, `api_key_id = -1` is `Chatroom`; user labels prefer the full name and fall back to email; app labels prefer the title, then origin URL, then `App #<id>` | ||
| - **Truncation**: when consuming output programmatically, check `metadata.truncated`. If `true`, the result was capped at `--limit` and is a *partial* dataset — raise `--limit` or paginate before reporting totals or rankings | ||
| - **Accounting note**: Server-tool billing rows are included in `total_usage` but excluded from `request_count`, request-based rates, and classification/dedup counts. Spend-per-request can therefore look inconsistent. | ||
|
|
||
| ### Cost Optimization Guidance | ||
|
|
||
|
|
@@ -147,6 +150,8 @@ When the user asks "How can I spend less?" or similar: | |
|
|
||
| To inspect specific generations or sessions from your analytics results, add `generation_id` or `session_id` as a dimension. Both are generations-only dimensions (31-day limit). `generation_id` returns the unique ID for each generation in the result set. `session_id` groups and filters sessionless requests as the literal `none`: the ClickHouse column defaults to an empty string, and the query builder coalesces it to `none`. Use `neq 'none'` to exclude sessionless requests; filtering on `''` matches nothing. | ||
|
|
||
| To break down requests by serving region, add `data_region` as a dimension. Values are `global`, `europe`, or `us`; it is generations-only (31-day limit), and rows predating region attribution report as `global`. | ||
|
|
||
| ```bash | ||
| npx tsx query-analytics.ts --metrics total_usage,tokens_total --dimensions generation_id --order-by total_usage --limit 10 | ||
| ``` | ||
|
|
||
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
🔍 data_region documented in two skills but missing from the schema skill's dimension list
This PR documents
data_regioninskills/openrouter-analytics-query/SKILL.md:97and inskills/openrouter-analytics/SKILL.md:153, but the schema-discovery skill's dimension catalog (skills/openrouter-analytics-schema/SKILL.md:139-146, "Limited to 31-day time ranges") and its non-enriched dimension list (skills/openrouter-analytics-schema/SKILL.md:236) were not updated. Since the schema skill is the one agents consult to learn which dimensions exist, the omission leaves the three skills inconsistent (session_idhas the same pre-existing gap). Worth confirming whether this is deliberate (e.g. covered by another open sync PR) or an oversight.Was this helpful? React with 👍 or 👎 to provide feedback.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Deliberate. The schema-skill entries for both dimensions are already open:
data_regionin #112,session_idin #100. Touching the same lines here would create a conflict between two sync PRs, so this PR covers only the two skills those PRs do not.Action needed: @jtcies, merging #112 and #100 (or this PR after them) is what closes the three-skill gap.