diff --git a/skills/openrouter-analytics-query/SKILL.md b/skills/openrouter-analytics-query/SKILL.md index f6b7446..d8b5f51 100644 --- a/skills/openrouter-analytics-query/SKILL.md +++ b/skills/openrouter-analytics-query/SKILL.md @@ -90,10 +90,11 @@ cd /scripts && npx tsx query-analytics.ts --met - Several dimensions are **label-resolved** in query results (returned as human-readable names), but filters must use the underlying ID: - `api_key_id` — numeric ID (from generation metadata) or 64-char SHA-256 hash (from `GET /api/v1/keys`). Hashes are auto-resolved to numeric IDs before querying. - `user` — Clerk user ID (e.g. `user_abc123`), not the display name/email shown in results. - - `workspace` — workspace UUID, not the workspace name shown in results. + - `workspace` — workspace UUID, not the workspace name shown in results; filtering or grouping by the account's default workspace also covers activity recorded before workspace resolution existed, which is attributed to that default workspace. - `app` — numeric app ID, not the app title shown in results. - - `model` — permaslug (e.g. `openai/gpt-4o`), not the display name. -- Other dimensions (`provider`, `origin`, `country`, `finish_reason`, `external_user`, etc.) are not enriched — filter values match what's returned in results. + - `model` — permaslug (e.g. `openai/gpt-4o`); both the filter value and returned value are the permaslug, not a display name. +- Other dimensions (`provider`, `origin`, `country`, `data_region`, `finish_reason`, `external_user`, etc.) are not enriched — filter values match what's returned in results. +- `data_region` values are `global`, `europe`, or `us`. It is a generations-only dimension (31-day limit), and rows predating region attribution report as `global`. ### Order By @@ -181,16 +182,18 @@ Classifier filters narrow results to generations matching specific classificatio | Field | Description | |---|---| -| `data.data` | Array of result rows. Each row has keys for requested metrics, dimensions, and `date__` (when granularity is set). For `classifier_dimensions` queries with a single `dimension_name`, a column is aliased to that name (e.g., `category`). With multiple names or no `dimension_names`, rows include `clf_dimension_name` and `clf_dimension_value` columns. | +| `data.data` | Array of result rows. Each row has keys for requested metrics, dimensions, and a source-dependent time-bucket field — `date__` for MV-backed queries or `created_at__` for raw generations/classification queries (detect either prefix). For `classifier_dimensions` queries with a single `dimension_name`, a column is aliased to that name (e.g., `category`). With multiple names or no `dimension_names`, rows include `clf_dimension_name` and `clf_dimension_value` columns. | | `data.metadata.query_time_ms` | Query execution time in milliseconds | | `data.metadata.row_count` | Number of rows returned | | `data.metadata.truncated` | `true` if results were truncated at the limit | | `data.cachedAt` | Unix timestamp (ms) when the result was cached. Present when the response was served from cache | -| `data.warnings` | Optional array of non-fatal warnings (e.g., unresolvable api_key_id hashes). The query still executes normally; these inform the caller about filter resolution issues. | +| `data.warnings` | Optional array of non-fatal warnings (e.g., unresolvable api_key_id hashes). An unresolvable label filter becomes a no-match sentinel; the query still executes normally, and the warning explains the filter resolution issue. | -> **Numeric types:** Count metrics (`request_count`, `tokens_*`, etc.) are returned as strings (`"1523"`). Cost and rate metrics (`total_usage`, `cache_hit_rate`, latency, throughput) are returned as numbers (`4.27`). Parse count values with `Number()` or `parseInt()` before arithmetic. +> **Numeric types:** Count metrics (`request_count`, `tokens_*`, etc.) are returned as strings (`"1523"`). Cost and rate metrics (`total_usage`, `blended_cost_per_million_tokens`, `cache_hit_rate`, latency, throughput) are returned as numbers (`4.27`). Parse count values with `Number()` or `parseInt()` before arithmetic. -> **Label resolution:** Dimensions `api_key_id`, `app`, `user`, and `workspace` return human-readable labels in data rows (key names, app titles, user names, workspace names), not raw IDs. +> **Accounting note:** Server-tool billing rows are included in `total_usage` but excluded from `request_count`, request-based rates, and classification/dedup counts. Spend-per-request can therefore look inconsistent. + +> **Label resolution:** Dimensions `api_key_id`, `app`, `user`, and `workspace` return human-readable labels in data rows (key names, app titles, user names, workspace names), not raw IDs. `app = -1` is `Unknown`, `api_key_id = -1` is `Chatroom`; user labels prefer the full name and fall back to email; app labels prefer the title, then origin URL, then `App #`. ## CLI Reference @@ -224,7 +227,7 @@ The CLI prints a single JSON object to **stdout** with two keys — `data` (the } ``` -A human-readable stats line (row count, query time, truncation/cache flags) is written to **stderr** for terminal use only. +A human-readable stats line (row count, query time, truncation/cache flags) is written to **stderr** for terminal use only. The CLI stdout intentionally omits the endpoint's optional `cachedAt` and `warnings` fields; use the direct API response when those fields are needed. > **When parsing output programmatically, always check `metadata.truncated`.** If `true`, the result was capped at `--limit` and is a *partial* dataset — increase `--limit` or paginate before reporting totals/rankings. Dimensions `api_key_id`, `user`, `app`, and `workspace` are already resolved to human-readable names in the data rows. @@ -330,12 +333,14 @@ Combine up to 2 dimensions for cross-tabulation: ## Time Range Behavior -Some metric/dimension combinations support time ranges up to **365 days** (with daily granularity), while others are limited to **31 days**. The server resolves this automatically based on the requested metrics and dimensions. +Some metric/dimension combinations support time ranges up to **367 days** (with daily granularity), while others are limited to **31 days**. The server resolves this automatically based on the requested metrics and dimensions. -Usage breakdown metrics follow the same pattern: `credits_usage`, `usage_upstream`, `usage_cache`, `usage_data`, `usage_web`, and `usage_upstream_web` support up to 365 days, while `openrouter_usage`, `byok_fees`, `usage_file`, `usage_upstream_file`, `usage_web_fetch`, and `usage_upstream_web_fetch` are limited to 31 days. +Usage breakdown metrics follow the same pattern: `credits_usage`, `usage_upstream`, `usage_cache`, `usage_data`, `usage_web`, and `usage_upstream_web` support up to 367 days, while `openrouter_usage`, `byok_fees`, `usage_file`, `usage_upstream_file`, `usage_web_fetch`, and `usage_upstream_web_fetch` are limited to 31 days. Classifier dimensions and classifier filters always force the 31-day time range limit. +A filter on a dimension not carried by the materialized views (for example, `data_region`) forces the query onto raw generations, so the 31-day limit applies even when the requested metrics and grouped dimensions would otherwise allow the longer daily-MV range. + If a query times out, try: - Narrowing the time range - Removing latency/throughput metrics diff --git a/skills/openrouter-analytics-schema/SKILL.md b/skills/openrouter-analytics-schema/SKILL.md index d05608b..be7fa46 100644 --- a/skills/openrouter-analytics-schema/SKILL.md +++ b/skills/openrouter-analytics-schema/SKILL.md @@ -60,30 +60,30 @@ Each metric has: ### Time Range Limits -Most volume and cost metrics support time ranges up to **365 days** with daily granularity. Latency/throughput metrics and some dimensions (`provider`, `origin`, `country`, `finish_reason`, `external_user`, `context_length_bucket`, `generation_id`) are limited to **31-day** time ranges. If a query times out, try narrowing the time range or removing latency/throughput metrics and per-generation dimensions. +Most volume and cost metrics support time ranges up to **367 days** with daily granularity. Latency/throughput metrics and some dimensions (`provider`, `origin`, `country`, `finish_reason`, `external_user`, `context_length_bucket`, `generation_id`) are limited to **31-day** time ranges. If a query times out, try narrowing the time range or removing latency/throughput metrics and per-generation dimensions. ### Metric Categories **Volume metrics** (how much): -- `request_count` — number of API requests (up to 365 days) -- `tokens_total`, `tokens_prompt`, `tokens_completion` — token counts (up to 365 days) -- `reasoning_tokens` — tokens used for extended thinking (up to 365 days) -- `cached_tokens` — tokens served from cache (up to 365 days) -- `byok_request_count` — number of BYOK requests (up to 365 days) +- `request_count` — number of API requests (up to 367 days) +- `tokens_total`, `tokens_prompt`, `tokens_completion` — token counts (up to 367 days) +- `reasoning_tokens` — tokens used for extended thinking (up to 367 days) +- `cached_tokens` — tokens served from cache (up to 367 days) +- `byok_request_count` — number of BYOK requests (up to 367 days) - `guardrail_invoked_count` — count of requests that triggered guardrails (31-day limit) - `response_cached_count` — count of responses served from cache (31-day limit) **Cost metrics** (how much money): -- `total_usage` — total cost in USD, including BYOK inference cost (up to 365 days). Computed as `sum(usage) + sum(byok_usage_inference)` so it reflects true spend for both credits and BYOK users. -- `byok_usage` — BYOK (bring your own key) inference cost in USD (up to 365 days) -- `credits_usage` — all charges billed to OpenRouter credits in USD, including BYOK platform fees (up to 365 days) +- `total_usage` — total cost in USD, including BYOK inference cost (up to 367 days). Computed as `sum(usage) + sum(byok_usage_inference)` so it reflects true spend for both credits and BYOK users. +- `byok_usage` — BYOK (bring your own key) inference cost in USD (up to 367 days) +- `credits_usage` — all charges billed to OpenRouter credits in USD, including BYOK platform fees (up to 367 days) - `openrouter_usage` — non-BYOK inference spend in USD; excludes requests made with user-provided keys (31-day limit) - `byok_fees` — BYOK platform fees in USD; the platform fee portion of `credits_usage` charged on BYOK requests (31-day limit). `credits_usage` includes both non-BYOK inference charges and these BYOK platform fees. -- `usage_upstream` — provider-side (upstream) cost in USD (up to 365 days) -- `usage_cache` — cache cost component in USD (up to 365 days) -- `usage_data` — data logging cost adjustment in USD; typically negative when a data logging discount applies (up to 365 days) -- `usage_web` — web search cost in USD (up to 365 days) -- `usage_upstream_web` — provider-side web search cost in USD (up to 365 days) +- `usage_upstream` — provider-side (upstream) cost in USD (up to 367 days) +- `usage_cache` — cache cost component in USD (up to 367 days) +- `usage_data` — data logging cost adjustment in USD; typically negative when a data logging discount applies (up to 367 days) +- `usage_web` — web search cost in USD (up to 367 days) +- `usage_upstream_web` — provider-side web search cost in USD (up to 367 days) - `usage_file` — file processing cost in USD (31-day limit) - `usage_upstream_file` — provider-side file processing cost in USD (31-day limit) - `usage_web_fetch` — web fetch cost in USD (31-day limit) @@ -98,6 +98,8 @@ Most volume and cost metrics support time ranges up to **365 days** with daily g - `guardrail_invoked_rate` — ratio of requests that triggered guardrails - `response_cached_rate` — ratio of responses served from cache +> **Accounting note:** Server-tool billing rows are included in `total_usage` but excluded from `request_count`, request-based rates, and classification/dedup counts. Spend-per-request can therefore look inconsistent. + ## Understanding Dimensions Each dimension has: @@ -126,6 +128,8 @@ All other dimensions (e.g., `model`, `provider`, `country`) are returned as-is w > Rows with an empty `user` value represent traffic not attributed to a specific org member (e.g., API keys created at the org level). +> Label fallbacks: `app = -1` is returned as `Unknown`, `api_key_id = -1` as `Chatroom`; user labels prefer the full name and fall back to email; app labels prefer the title, then origin URL, then `App #`. + ### Dimension Categories **Available with all time ranges:** @@ -210,14 +214,14 @@ Use this guide to translate natural-language questions into the right metric/dim | "Usage by country" | `request_count` | `country` | 31-day limit | | "How can I save money?" | `total_usage`, `cache_hit_rate`, `tokens_total` | `model` | See cost optimization in `openrouter-analytics` skill | | "Show me individual requests" | `total_usage`, `tokens_total` | `generation_id` | 31-day limit. Use returned IDs with `openrouter-generations` skill for full metadata and content | -| "How much BYOK spend?" | `byok_usage` | `model` | Up to 365 days | -| "BYOK vs credits split?" | `byok_usage`, `credits_usage` | — | Both up to 365 days | +| "How much BYOK spend?" | `byok_usage` | `model` | Up to 367 days | +| "BYOK vs credits split?" | `byok_usage`, `credits_usage` | — | Both up to 367 days | | "BYOK platform fees?" | `byok_fees` | `model` | 31-day limit | | "Non-BYOK inference spend?" | `openrouter_usage` | `model` | 31-day limit | | "How many guardrail triggers?" | `guardrail_invoked_count`, `guardrail_invoked_rate` | `model` | 31-day limit | | "How many cached responses?" | `response_cached_count`, `response_cached_rate` | `model` | 31-day limit | -| "Where does my spend go?" | `usage_upstream`, `usage_cache`, `usage_data` | — | Full cost breakdown (up to 365 days) | -| "Web search costs?" | `usage_web`, `usage_upstream_web` | `model` | Up to 365 days | +| "Where does my spend go?" | `usage_upstream`, `usage_cache`, `usage_data` | — | Full cost breakdown (up to 367 days) | +| "Web search costs?" | `usage_web`, `usage_upstream_web` | `model` | Up to 367 days | | "File processing costs?" | `usage_file`, `usage_upstream_file` | `model` | 31-day limit | | "Web fetch costs?" | `usage_web_fetch`, `usage_upstream_web_fetch` | `model` | 31-day limit | @@ -243,7 +247,7 @@ Other dimensions (`provider`, `origin`, `country`, `finish_reason`, `external_us - Maximum 10 classifier filters per query - Maximum 10,000 rows returned per query (default 1,000) - `group_limit` (1–10,000): controls max rows per dimension combination. Auto-computed on time-series queries with dimensions to guarantee full time-window coverage. Set explicitly to cap per-group rows (e.g., top N per model per day). -- Most volume/cost metrics: up to 365 days with daily granularity +- Most volume/cost metrics: up to 367 days with daily granularity - Latency/throughput metrics and per-generation dimensions: up to 31 days - Classifier dimensions/filters: always limited to 31 days - Minute granularity: only available when the time window is ≤ 3 hours diff --git a/skills/openrouter-analytics/SKILL.md b/skills/openrouter-analytics/SKILL.md index 2b7cdd3..446e48a 100644 --- a/skills/openrouter-analytics/SKILL.md +++ b/skills/openrouter-analytics/SKILL.md @@ -122,13 +122,16 @@ The query endpoint returns an array of data rows. Each row is a flat object with When interpreting results for the user: - **Spend metrics** (`total_usage`, `credits_usage`, `openrouter_usage`, `byok_usage`, `byok_fees`, `usage_upstream`, `usage_cache`, `usage_web`, `usage_upstream_web`, `usage_file`, `usage_upstream_file`, `usage_web_fetch`, `usage_upstream_web_fetch`) are in USD. `total_usage` includes BYOK inference cost. `usage_data` is typically negative (a data logging discount) +- **Blended cost** (`blended_cost_per_million_tokens`) is the blended inference cost per 1M tokens, including applicable BYOK inference usage - **Token counts** (`tokens_total`, `tokens_prompt`, `tokens_completion`) are in native model tokens - **Latency** (`avg_latency`, `p50_latency`, etc.) is in milliseconds - **Rates** (`cache_hit_rate`) are 0–1 ratios - **Throughput** (`avg_throughput`) is tokens per second -- When `granularity` is set, rows include a `date__` field for the time bucket (e.g., `date__day`, `date__hour`, `date__month`) +- When `granularity` is set, rows include a source-dependent time-bucket field: `date__` for MV-backed queries or `created_at__` for raw generations/classification queries (detect either prefix) - **Label resolution**: dimensions `api_key_id`, `app`, `user`, and `workspace` have their raw IDs replaced with human-readable names (key name, app title, user name, workspace name) directly in the data rows; `generation_id` and `session_id` return raw values +- **Label fallbacks**: `app = -1` is `Unknown`, `api_key_id = -1` is `Chatroom`; user labels prefer the full name and fall back to email; app labels prefer the title, then origin URL, then `App #` - **Truncation**: when consuming output programmatically, check `metadata.truncated`. If `true`, the result was capped at `--limit` and is a *partial* dataset — raise `--limit` or paginate before reporting totals or rankings +- **Accounting note**: Server-tool billing rows are included in `total_usage` but excluded from `request_count`, request-based rates, and classification/dedup counts. Spend-per-request can therefore look inconsistent. ### Cost Optimization Guidance @@ -147,6 +150,8 @@ When the user asks "How can I spend less?" or similar: To inspect specific generations or sessions from your analytics results, add `generation_id` or `session_id` as a dimension. Both are generations-only dimensions (31-day limit). `generation_id` returns the unique ID for each generation in the result set. `session_id` groups and filters sessionless requests as the literal `none`: the ClickHouse column defaults to an empty string, and the query builder coalesces it to `none`. Use `neq 'none'` to exclude sessionless requests; filtering on `''` matches nothing. +To break down requests by serving region, add `data_region` as a dimension. Values are `global`, `europe`, or `us`; it is generations-only (31-day limit), and rows predating region attribution report as `global`. + ```bash npx tsx query-analytics.ts --metrics total_usage,tokens_total --dimensions generation_id --order-by total_usage --limit 10 ```