Skip to content
25 changes: 15 additions & 10 deletions skills/openrouter-analytics-query/SKILL.md
Original file line number Diff line number Diff line change
Expand Up @@ -90,10 +90,11 @@ cd <openrouter-analytics-skill-path>/scripts && npx tsx query-analytics.ts --met
- Several dimensions are **label-resolved** in query results (returned as human-readable names), but filters must use the underlying ID:
- `api_key_id` — numeric ID (from generation metadata) or 64-char SHA-256 hash (from `GET /api/v1/keys`). Hashes are auto-resolved to numeric IDs before querying.
- `user` — Clerk user ID (e.g. `user_abc123`), not the display name/email shown in results.
- `workspace` — workspace UUID, not the workspace name shown in results.
- `workspace` — workspace UUID, not the workspace name shown in results; filtering or grouping by the account's default workspace also covers activity recorded before workspace resolution existed, which is attributed to that default workspace.
- `app` — numeric app ID, not the app title shown in results.
- `model` — permaslug (e.g. `openai/gpt-4o`), not the display name.
- Other dimensions (`provider`, `origin`, `country`, `finish_reason`, `external_user`, etc.) are not enriched — filter values match what's returned in results.
- `model` — permaslug (e.g. `openai/gpt-4o`); both the filter value and returned value are the permaslug, not a display name.
- Other dimensions (`provider`, `origin`, `country`, `data_region`, `finish_reason`, `external_user`, etc.) are not enriched — filter values match what's returned in results.
- `data_region` values are `global`, `europe`, or `us`. It is a generations-only dimension (31-day limit), and rows predating region attribution report as `global`.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔍 data_region documented in two skills but missing from the schema skill's dimension list

This PR documents data_region in skills/openrouter-analytics-query/SKILL.md:97 and in skills/openrouter-analytics/SKILL.md:153, but the schema-discovery skill's dimension catalog (skills/openrouter-analytics-schema/SKILL.md:139-146, "Limited to 31-day time ranges") and its non-enriched dimension list (skills/openrouter-analytics-schema/SKILL.md:236) were not updated. Since the schema skill is the one agents consult to learn which dimensions exist, the omission leaves the three skills inconsistent (session_id has the same pre-existing gap). Worth confirming whether this is deliberate (e.g. covered by another open sync PR) or an oversight.

Open in Devin Review

Was this helpful? React with 👍 or 👎 to provide feedback.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Deliberate. The schema-skill entries for both dimensions are already open: data_region in #112, session_id in #100. Touching the same lines here would create a conflict between two sync PRs, so this PR covers only the two skills those PRs do not.

Action needed: @jtcies, merging #112 and #100 (or this PR after them) is what closes the three-skill gap.


### Order By

Expand Down Expand Up @@ -181,16 +182,18 @@ Classifier filters narrow results to generations matching specific classificatio

| Field | Description |
|---|---|
| `data.data` | Array of result rows. Each row has keys for requested metrics, dimensions, and `date__<granularity>` (when granularity is set). For `classifier_dimensions` queries with a single `dimension_name`, a column is aliased to that name (e.g., `category`). With multiple names or no `dimension_names`, rows include `clf_dimension_name` and `clf_dimension_value` columns. |
| `data.data` | Array of result rows. Each row has keys for requested metrics, dimensions, and a source-dependent time-bucket field — `date__<granularity>` for MV-backed queries or `created_at__<granularity>` for raw generations/classification queries (detect either prefix). For `classifier_dimensions` queries with a single `dimension_name`, a column is aliased to that name (e.g., `category`). With multiple names or no `dimension_names`, rows include `clf_dimension_name` and `clf_dimension_value` columns. |
| `data.metadata.query_time_ms` | Query execution time in milliseconds |
| `data.metadata.row_count` | Number of rows returned |
| `data.metadata.truncated` | `true` if results were truncated at the limit |
| `data.cachedAt` | Unix timestamp (ms) when the result was cached. Present when the response was served from cache |
| `data.warnings` | Optional array of non-fatal warnings (e.g., unresolvable api_key_id hashes). The query still executes normally; these inform the caller about filter resolution issues. |
| `data.warnings` | Optional array of non-fatal warnings (e.g., unresolvable api_key_id hashes). An unresolvable label filter becomes a no-match sentinel; the query still executes normally, and the warning explains the filter resolution issue. |

> **Numeric types:** Count metrics (`request_count`, `tokens_*`, etc.) are returned as strings (`"1523"`). Cost and rate metrics (`total_usage`, `cache_hit_rate`, latency, throughput) are returned as numbers (`4.27`). Parse count values with `Number()` or `parseInt()` before arithmetic.
> **Numeric types:** Count metrics (`request_count`, `tokens_*`, etc.) are returned as strings (`"1523"`). Cost and rate metrics (`total_usage`, `blended_cost_per_million_tokens`, `cache_hit_rate`, latency, throughput) are returned as numbers (`4.27`). Parse count values with `Number()` or `parseInt()` before arithmetic.

> **Label resolution:** Dimensions `api_key_id`, `app`, `user`, and `workspace` return human-readable labels in data rows (key names, app titles, user names, workspace names), not raw IDs.
> **Accounting note:** Server-tool billing rows are included in `total_usage` but excluded from `request_count`, request-based rates, and classification/dedup counts. Spend-per-request can therefore look inconsistent.

> **Label resolution:** Dimensions `api_key_id`, `app`, `user`, and `workspace` return human-readable labels in data rows (key names, app titles, user names, workspace names), not raw IDs. `app = -1` is `Unknown`, `api_key_id = -1` is `Chatroom`; user labels prefer the full name and fall back to email; app labels prefer the title, then origin URL, then `App #<id>`.

## CLI Reference

Expand Down Expand Up @@ -224,7 +227,7 @@ The CLI prints a single JSON object to **stdout** with two keys — `data` (the
}
```

A human-readable stats line (row count, query time, truncation/cache flags) is written to **stderr** for terminal use only.
A human-readable stats line (row count, query time, truncation/cache flags) is written to **stderr** for terminal use only. The CLI stdout intentionally omits the endpoint's optional `cachedAt` and `warnings` fields; use the direct API response when those fields are needed.

> **When parsing output programmatically, always check `metadata.truncated`.** If `true`, the result was capped at `--limit` and is a *partial* dataset — increase `--limit` or paginate before reporting totals/rankings. Dimensions `api_key_id`, `user`, `app`, and `workspace` are already resolved to human-readable names in the data rows.

Expand Down Expand Up @@ -330,12 +333,14 @@ Combine up to 2 dimensions for cross-tabulation:

## Time Range Behavior

Some metric/dimension combinations support time ranges up to **365 days** (with daily granularity), while others are limited to **31 days**. The server resolves this automatically based on the requested metrics and dimensions.
Some metric/dimension combinations support time ranges up to **367 days** (with daily granularity), while others are limited to **31 days**. The server resolves this automatically based on the requested metrics and dimensions.

Usage breakdown metrics follow the same pattern: `credits_usage`, `usage_upstream`, `usage_cache`, `usage_data`, `usage_web`, and `usage_upstream_web` support up to 365 days, while `openrouter_usage`, `byok_fees`, `usage_file`, `usage_upstream_file`, `usage_web_fetch`, and `usage_upstream_web_fetch` are limited to 31 days.
Usage breakdown metrics follow the same pattern: `credits_usage`, `usage_upstream`, `usage_cache`, `usage_data`, `usage_web`, and `usage_upstream_web` support up to 367 days, while `openrouter_usage`, `byok_fees`, `usage_file`, `usage_upstream_file`, `usage_web_fetch`, and `usage_upstream_web_fetch` are limited to 31 days.

Classifier dimensions and classifier filters always force the 31-day time range limit.

A filter on a dimension not carried by the materialized views (for example, `data_region`) forces the query onto raw generations, so the 31-day limit applies even when the requested metrics and grouped dimensions would otherwise allow the longer daily-MV range.

If a query times out, try:
- Narrowing the time range
- Removing latency/throughput metrics
Expand Down
42 changes: 23 additions & 19 deletions skills/openrouter-analytics-schema/SKILL.md
Original file line number Diff line number Diff line change
Expand Up @@ -60,30 +60,30 @@ Each metric has:

### Time Range Limits

Most volume and cost metrics support time ranges up to **365 days** with daily granularity. Latency/throughput metrics and some dimensions (`provider`, `origin`, `country`, `finish_reason`, `external_user`, `context_length_bucket`, `generation_id`) are limited to **31-day** time ranges. If a query times out, try narrowing the time range or removing latency/throughput metrics and per-generation dimensions.
Most volume and cost metrics support time ranges up to **367 days** with daily granularity. Latency/throughput metrics and some dimensions (`provider`, `origin`, `country`, `finish_reason`, `external_user`, `context_length_bucket`, `generation_id`) are limited to **31-day** time ranges. If a query times out, try narrowing the time range or removing latency/throughput metrics and per-generation dimensions.
Comment thread
devin-ai-integration[bot] marked this conversation as resolved.

### Metric Categories

**Volume metrics** (how much):
- `request_count` — number of API requests (up to 365 days)
- `tokens_total`, `tokens_prompt`, `tokens_completion` — token counts (up to 365 days)
- `reasoning_tokens` — tokens used for extended thinking (up to 365 days)
- `cached_tokens` — tokens served from cache (up to 365 days)
- `byok_request_count` — number of BYOK requests (up to 365 days)
- `request_count` — number of API requests (up to 367 days)
- `tokens_total`, `tokens_prompt`, `tokens_completion` — token counts (up to 367 days)
- `reasoning_tokens` — tokens used for extended thinking (up to 367 days)
- `cached_tokens` — tokens served from cache (up to 367 days)
- `byok_request_count` — number of BYOK requests (up to 367 days)
- `guardrail_invoked_count` — count of requests that triggered guardrails (31-day limit)
- `response_cached_count` — count of responses served from cache (31-day limit)

**Cost metrics** (how much money):
- `total_usage` — total cost in USD, including BYOK inference cost (up to 365 days). Computed as `sum(usage) + sum(byok_usage_inference)` so it reflects true spend for both credits and BYOK users.
- `byok_usage` — BYOK (bring your own key) inference cost in USD (up to 365 days)
- `credits_usage` — all charges billed to OpenRouter credits in USD, including BYOK platform fees (up to 365 days)
- `total_usage` — total cost in USD, including BYOK inference cost (up to 367 days). Computed as `sum(usage) + sum(byok_usage_inference)` so it reflects true spend for both credits and BYOK users.
- `byok_usage` — BYOK (bring your own key) inference cost in USD (up to 367 days)
- `credits_usage` — all charges billed to OpenRouter credits in USD, including BYOK platform fees (up to 367 days)
- `openrouter_usage` — non-BYOK inference spend in USD; excludes requests made with user-provided keys (31-day limit)
- `byok_fees` — BYOK platform fees in USD; the platform fee portion of `credits_usage` charged on BYOK requests (31-day limit). `credits_usage` includes both non-BYOK inference charges and these BYOK platform fees.
- `usage_upstream` — provider-side (upstream) cost in USD (up to 365 days)
- `usage_cache` — cache cost component in USD (up to 365 days)
- `usage_data` — data logging cost adjustment in USD; typically negative when a data logging discount applies (up to 365 days)
- `usage_web` — web search cost in USD (up to 365 days)
- `usage_upstream_web` — provider-side web search cost in USD (up to 365 days)
- `usage_upstream` — provider-side (upstream) cost in USD (up to 367 days)
- `usage_cache` — cache cost component in USD (up to 367 days)
- `usage_data` — data logging cost adjustment in USD; typically negative when a data logging discount applies (up to 367 days)
- `usage_web` — web search cost in USD (up to 367 days)
- `usage_upstream_web` — provider-side web search cost in USD (up to 367 days)
- `usage_file` — file processing cost in USD (31-day limit)
- `usage_upstream_file` — provider-side file processing cost in USD (31-day limit)
- `usage_web_fetch` — web fetch cost in USD (31-day limit)
Expand All @@ -98,6 +98,8 @@ Most volume and cost metrics support time ranges up to **365 days** with daily g
- `guardrail_invoked_rate` — ratio of requests that triggered guardrails
- `response_cached_rate` — ratio of responses served from cache

> **Accounting note:** Server-tool billing rows are included in `total_usage` but excluded from `request_count`, request-based rates, and classification/dedup counts. Spend-per-request can therefore look inconsistent.

## Understanding Dimensions

Each dimension has:
Expand Down Expand Up @@ -126,6 +128,8 @@ All other dimensions (e.g., `model`, `provider`, `country`) are returned as-is w

> Rows with an empty `user` value represent traffic not attributed to a specific org member (e.g., API keys created at the org level).

> Label fallbacks: `app = -1` is returned as `Unknown`, `api_key_id = -1` as `Chatroom`; user labels prefer the full name and fall back to email; app labels prefer the title, then origin URL, then `App #<id>`.

### Dimension Categories

**Available with all time ranges:**
Expand Down Expand Up @@ -210,14 +214,14 @@ Use this guide to translate natural-language questions into the right metric/dim
| "Usage by country" | `request_count` | `country` | 31-day limit |
| "How can I save money?" | `total_usage`, `cache_hit_rate`, `tokens_total` | `model` | See cost optimization in `openrouter-analytics` skill |
| "Show me individual requests" | `total_usage`, `tokens_total` | `generation_id` | 31-day limit. Use returned IDs with `openrouter-generations` skill for full metadata and content |
| "How much BYOK spend?" | `byok_usage` | `model` | Up to 365 days |
| "BYOK vs credits split?" | `byok_usage`, `credits_usage` | — | Both up to 365 days |
| "How much BYOK spend?" | `byok_usage` | `model` | Up to 367 days |
| "BYOK vs credits split?" | `byok_usage`, `credits_usage` | — | Both up to 367 days |
| "BYOK platform fees?" | `byok_fees` | `model` | 31-day limit |
| "Non-BYOK inference spend?" | `openrouter_usage` | `model` | 31-day limit |
| "How many guardrail triggers?" | `guardrail_invoked_count`, `guardrail_invoked_rate` | `model` | 31-day limit |
| "How many cached responses?" | `response_cached_count`, `response_cached_rate` | `model` | 31-day limit |
| "Where does my spend go?" | `usage_upstream`, `usage_cache`, `usage_data` | — | Full cost breakdown (up to 365 days) |
| "Web search costs?" | `usage_web`, `usage_upstream_web` | `model` | Up to 365 days |
| "Where does my spend go?" | `usage_upstream`, `usage_cache`, `usage_data` | — | Full cost breakdown (up to 367 days) |
| "Web search costs?" | `usage_web`, `usage_upstream_web` | `model` | Up to 367 days |
| "File processing costs?" | `usage_file`, `usage_upstream_file` | `model` | 31-day limit |
| "Web fetch costs?" | `usage_web_fetch`, `usage_upstream_web_fetch` | `model` | 31-day limit |

Expand All @@ -243,7 +247,7 @@ Other dimensions (`provider`, `origin`, `country`, `finish_reason`, `external_us
- Maximum 10 classifier filters per query
- Maximum 10,000 rows returned per query (default 1,000)
- `group_limit` (1–10,000): controls max rows per dimension combination. Auto-computed on time-series queries with dimensions to guarantee full time-window coverage. Set explicitly to cap per-group rows (e.g., top N per model per day).
- Most volume/cost metrics: up to 365 days with daily granularity
- Most volume/cost metrics: up to 367 days with daily granularity
- Latency/throughput metrics and per-generation dimensions: up to 31 days
- Classifier dimensions/filters: always limited to 31 days
- Minute granularity: only available when the time window is ≤ 3 hours
Expand Down
7 changes: 6 additions & 1 deletion skills/openrouter-analytics/SKILL.md
Original file line number Diff line number Diff line change
Expand Up @@ -122,13 +122,16 @@ The query endpoint returns an array of data rows. Each row is a flat object with

When interpreting results for the user:
- **Spend metrics** (`total_usage`, `credits_usage`, `openrouter_usage`, `byok_usage`, `byok_fees`, `usage_upstream`, `usage_cache`, `usage_web`, `usage_upstream_web`, `usage_file`, `usage_upstream_file`, `usage_web_fetch`, `usage_upstream_web_fetch`) are in USD. `total_usage` includes BYOK inference cost. `usage_data` is typically negative (a data logging discount)
- **Blended cost** (`blended_cost_per_million_tokens`) is the blended inference cost per 1M tokens, including applicable BYOK inference usage

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔍 blended_cost_per_million_tokens described only in two of three skills

The metric is added to the interpretation list here and to the numeric-types note in skills/openrouter-analytics-query/SKILL.md:192, but it is absent from the schema skill's cost-metric catalog (skills/openrouter-analytics-schema/SKILL.md:76-90). The PR description states this is intentional because open PR #88 covers the schema entry; if that PR does not land, agents discovering metrics through the schema skill will not know the metric exists or its 31-day/367-day range class.

Open in Devin Review

Was this helpful? React with 👍 or 👎 to provide feedback.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Correct, and the risk is real but small. The schema-skill catalog entry lives in #88 for the same conflict-avoidance reason. If #88 is closed rather than merged, the metric should be added here instead.

Action needed: @jtcies, merge or close #88 so the schema catalog does not stay behind.

- **Token counts** (`tokens_total`, `tokens_prompt`, `tokens_completion`) are in native model tokens
- **Latency** (`avg_latency`, `p50_latency`, etc.) is in milliseconds
- **Rates** (`cache_hit_rate`) are 0–1 ratios
- **Throughput** (`avg_throughput`) is tokens per second
- When `granularity` is set, rows include a `date__<granularity>` field for the time bucket (e.g., `date__day`, `date__hour`, `date__month`)
- When `granularity` is set, rows include a source-dependent time-bucket field: `date__<granularity>` for MV-backed queries or `created_at__<granularity>` for raw generations/classification queries (detect either prefix)
- **Label resolution**: dimensions `api_key_id`, `app`, `user`, and `workspace` have their raw IDs replaced with human-readable names (key name, app title, user name, workspace name) directly in the data rows; `generation_id` and `session_id` return raw values
- **Label fallbacks**: `app = -1` is `Unknown`, `api_key_id = -1` is `Chatroom`; user labels prefer the full name and fall back to email; app labels prefer the title, then origin URL, then `App #<id>`
- **Truncation**: when consuming output programmatically, check `metadata.truncated`. If `true`, the result was capped at `--limit` and is a *partial* dataset — raise `--limit` or paginate before reporting totals or rankings
- **Accounting note**: Server-tool billing rows are included in `total_usage` but excluded from `request_count`, request-based rates, and classification/dedup counts. Spend-per-request can therefore look inconsistent.

### Cost Optimization Guidance

Expand All @@ -147,6 +150,8 @@ When the user asks "How can I spend less?" or similar:

To inspect specific generations or sessions from your analytics results, add `generation_id` or `session_id` as a dimension. Both are generations-only dimensions (31-day limit). `generation_id` returns the unique ID for each generation in the result set. `session_id` groups and filters sessionless requests as the literal `none`: the ClickHouse column defaults to an empty string, and the query builder coalesces it to `none`. Use `neq 'none'` to exclude sessionless requests; filtering on `''` matches nothing.

To break down requests by serving region, add `data_region` as a dimension. Values are `global`, `europe`, or `us`; it is generations-only (31-day limit), and rows predating region attribution report as `global`.

```bash
npx tsx query-analytics.ts --metrics total_usage,tokens_total --dimensions generation_id --order-by total_usage --limit 10
```
Expand Down