Skip to content

docs(analytics): server-tool accounting, 367-day cap, region + query behavior - #94

Open
jtcies wants to merge 6 commits into
mainfrom
devin/1784556350-weekly-analytics-sync
Open

docs(analytics): server-tool accounting, 367-day cap, region + query behavior#94
jtcies wants to merge 6 commits into
mainfrom
devin/1784556350-weekly-analytics-sync

Conversation

@jtcies

@jtcies jtcies commented Jul 20, 2026

Copy link
Copy Markdown
Contributor

Summary

Rolling weekly analytics skills sync. Each run appends the query-builder drift that no other open sync PR covers.

2026-08-03 run — drift from this week's packages/clickhouse/analytics/ commits, plus two pre-existing factual errors:

  • data_region dimension (openrouter-web #31199): documented in openrouter-analytics-query and openrouter-analytics. Generations-only, so it carries the 31-day cap; values global / europe / us; rows predating region attribution report as global. The schema-skill entry is intentionally left to open PR docs(openrouter-analytics-schema): sync with openrouter-web changes #112.
  • Generations-only filters shrink the range cap: a filter on a dimension the MVs don't carry (e.g. data_region) routes the query to raw generations, so the 31-day cap applies even when the requested metrics and grouped dimensions would allow the 367-day daily-MV range (trends/effective-range-cap.ts).
  • Workspace legacy-UUID folding (openrouter-web #30202, ENT-1724): filtering or grouping by the account's default workspace also covers activity recorded before workspace resolution existed. Wording mirrors the public OpenAPI filter-value description so the skill and the API agree.
  • model is not label-resolved (fix): the filter-value list implied results carry a model display name. The public query route deliberately excludes model from label substitution because the permaslug is the stable public identifier, so both the filter value and the returned value are the permaslug.
  • Time-bucket field name is source-dependent (fix): skills claimed rows always carry date__<granularity>. TIME_CONFIG_BY_SOURCE uses the date alias only for MV sources; raw generations and classifications use created_at, so rows can carry created_at__<granularity>. Readers are told to detect either prefix.

2026-07-20 run (earlier commits on this branch):

  • Server-tool accounting (openrouter-web #23325): server-tool billing rows are included in total_usage but excluded from request_count, request-based rates, and classification/dedup counts, so spend-per-request can look inconsistent.
  • blended_cost_per_million_tokens (openrouter-web #28910) added to the metric-interpretation list and the numeric-types note. The schema-skill entry is covered by open PR docs(openrouter-analytics-schema): sync with openrouter-web changes #88.
  • 365 → 367 day range cap: ANALYTICS_MAX_RANGE_MS_DAILY_MV = 367 * 24 * 60 * 60 * 1000 (1 year plus a 2-day buffer so the "Past 1 Year" preset survives leap days).
  • Label fallbacks: app = -1Unknown, api_key_id = -1Chatroom, user labels prefer full name then email, app labels prefer title → origin URL → App #<id>. Also notes that an unresolvable label filter becomes a no-match sentinel and that CLI stdout omits cachedAt / warnings.

Deliberately out of scope: session_id in the schema skill (open PR #100), the user / user_email split and order_by request_count (open PRs #90/#91/#92), and per-metric MV-vs-generations availability tables, which are internal detail the meta endpoint does not expose.

Link to Devin session: https://openrouter.devinenterprise.com/sessions/25bbc687de7e4775a703b2f930b09c0f
Requested by: @jtcies


Open in Devin Review

Co-Authored-By: Joseph Ciesielski <joseph.ciesielski@openrouter.ai>
@jtcies jtcies self-assigned this Jul 20, 2026
@devin-ai-integration

Copy link
Copy Markdown
Contributor
Original prompt from Joseph

# Weekly Analytics Skills Sync

You are responsible for keeping the analytics skills in OpenRouterTeam/skills in sync with the analytics query builder in OpenRouterTeam/openrouter-web. Post your findings to the Slack channel #proj-activity-observability (C0AKMN11K7G).

#``# Step 1: Check for recent query builder changes

Clone both repos:

  • OpenRouterTeam/openrouter-web (query builder source of truth)
  • OpenRouterTeam/skills (analytics skills that document the query builder)

Check git log for the last 7 days of changes in the query builder directory:

cd openrouter-web
git log --since="7 days ago" --oneline -- packages/clickhouse/analytics/

Also check the analytics route handlers:

git log --since="7 days ago" --oneline -- services/cfw-api/src/routes/analytics/

#``# Step 2: Read the current query builder state

Read these files to understand the current state of the query builder:

  • packages/clickhouse/analytics/schemas.ts — MetricNameSchema, DimensionNameSchema, Granularity, FilterOperator definitions
  • packages/clickhouse/analytics/metric-registry.ts — full metric definitions (name, displayLabel, availability, expressions, isRate, displayFormat)
  • packages/clickhouse/analytics/dimension-registry.ts — full dimension definitions (name, displayLabel, mvColumn, generationsColumn, availableIn, clickhouseType)
  • packages/clickhouse/analytics/operator-registry.ts — filter operator definitions
  • packages/clickhouse/analytics/table-resolver.ts — table resolution logic
  • packages/clickhouse/analytics/filter-builder.ts — filter building logic
  • packages/clickhouse/analytics/enrichment/ — label resolution (which dimensions get labels)
  • packages/clickhouse/analytics/trends/ — trends/leaderboard capabilities
  • services/cfw-api/src/routes/analytics/get-meta.ts — what the meta endpoint exposes

#``# Step 3: Read the current analytics skills

Read these skill files:

  • skills/openrouter-analytics/SKILL.md — main analytics skill
  • ... (2986 chars truncated...)

@devin-ai-integration

Copy link
Copy Markdown
Contributor

🤖 Devin AI Engineer

I'll be helping with this pull request! Here's what you should know:

✅ I will automatically:

  • Address comments on this PR that start with 'DevinAI' or '@devin'.
  • Look at CI failures and help fix them

Note: I can only respond to comments from users who have write access to this repository.

⚙️ Control Options:

  • Disable automatic comment, CI, and merge conflict monitoring

@devin-ai-integration devin-ai-integration Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ Devin Review: No Issues Found

Devin Review analyzed this PR and found no bugs or issues to report.

Open in Devin Review

Co-Authored-By: Joseph Ciesielski <joseph.ciesielski@openrouter.ai>
@devin-ai-integration devin-ai-integration Bot changed the title docs(analytics): document server-tool accounting and blended cost metric docs(analytics): server-tool accounting, 367-day cap, label fallbacks Jul 27, 2026
devin-ai-integration[bot]

This comment was marked as resolved.

devin-ai-integration Bot and others added 4 commits July 27, 2026 17:42
…backs)

Co-Authored-By: Joseph Ciesielski <joseph.ciesielski@openrouter.ai>
Co-Authored-By: Joseph Ciesielski <joseph.ciesielski@openrouter.ai>
Co-Authored-By: Joseph Ciesielski <joseph.ciesielski@openrouter.ai>
@devin-ai-integration devin-ai-integration Bot changed the title docs(analytics): server-tool accounting, 367-day cap, label fallbacks docs(analytics): server-tool accounting, 367-day cap, region + query behavior Aug 3, 2026

@devin-ai-integration devin-ai-integration Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Devin Review found 2 new potential issues.

Open in Devin Review

- Other dimensions (`provider`, `origin`, `country`, `finish_reason`, `external_user`, etc.) are not enriched — filter values match what's returned in results.
- `model` — permaslug (e.g. `openai/gpt-4o`); both the filter value and returned value are the permaslug, not a display name.
- Other dimensions (`provider`, `origin`, `country`, `data_region`, `finish_reason`, `external_user`, etc.) are not enriched — filter values match what's returned in results.
- `data_region` values are `global`, `europe`, or `us`. It is a generations-only dimension (31-day limit), and rows predating region attribution report as `global`.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔍 data_region documented in two skills but missing from the schema skill's dimension list

This PR documents data_region in skills/openrouter-analytics-query/SKILL.md:97 and in skills/openrouter-analytics/SKILL.md:153, but the schema-discovery skill's dimension catalog (skills/openrouter-analytics-schema/SKILL.md:139-146, "Limited to 31-day time ranges") and its non-enriched dimension list (skills/openrouter-analytics-schema/SKILL.md:236) were not updated. Since the schema skill is the one agents consult to learn which dimensions exist, the omission leaves the three skills inconsistent (session_id has the same pre-existing gap). Worth confirming whether this is deliberate (e.g. covered by another open sync PR) or an oversight.

Open in Devin Review

Was this helpful? React with 👍 or 👎 to provide feedback.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Deliberate. The schema-skill entries for both dimensions are already open: data_region in #112, session_id in #100. Touching the same lines here would create a conflict between two sync PRs, so this PR covers only the two skills those PRs do not.

Action needed: @jtcies, merging #112 and #100 (or this PR after them) is what closes the three-skill gap.


When interpreting results for the user:
- **Spend metrics** (`total_usage`, `credits_usage`, `openrouter_usage`, `byok_usage`, `byok_fees`, `usage_upstream`, `usage_cache`, `usage_web`, `usage_upstream_web`, `usage_file`, `usage_upstream_file`, `usage_web_fetch`, `usage_upstream_web_fetch`) are in USD. `total_usage` includes BYOK inference cost. `usage_data` is typically negative (a data logging discount)
- **Blended cost** (`blended_cost_per_million_tokens`) is the blended inference cost per 1M tokens, including applicable BYOK inference usage

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔍 blended_cost_per_million_tokens described only in two of three skills

The metric is added to the interpretation list here and to the numeric-types note in skills/openrouter-analytics-query/SKILL.md:192, but it is absent from the schema skill's cost-metric catalog (skills/openrouter-analytics-schema/SKILL.md:76-90). The PR description states this is intentional because open PR #88 covers the schema entry; if that PR does not land, agents discovering metrics through the schema skill will not know the metric exists or its 31-day/367-day range class.

Open in Devin Review

Was this helpful? React with 👍 or 👎 to provide feedback.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Correct, and the risk is real but small. The schema-skill catalog entry lives in #88 for the same conflict-avoidance reason. If #88 is closed rather than merged, the metric should be added here instead.

Action needed: @jtcies, merge or close #88 so the schema catalog does not stay behind.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant