feat(telemetry): RPC usage metrics — raw JSON-RPC request counts per node/method

[Bojan131]

Why

A node burned ~$200 of RPC credits in a day and nothing measured it. dkg.chain.rpc.total (PR #1317) counts logical chain operations, but RPC providers bill per raw JSON-RPC request — one logical op fans out into several raw requests (a tx populate alone issues eth_estimateGas + eth_getTransactionCount + eth_chainId + fee reads). Milestone 1 (manager): a usable view of RPC usage for the whole team, with limited metrics we can be sure are computed correctly.

What

• Exact billing-unit counting — CountingJsonRpcProvider hooks ethers' _send() (the actual wire dispatch; send() is bypassed by ethers' network detection) and counts every outgoing JSON-RPC request per method. With the adapter's batchMaxCount: 1, one entry == one HTTP request.
• Visible via the log pipeline already being deployed — every 60s the daemon logs one rpc_usage method=<m> count=<n> window_s=60 chain=<id> logfmt line per method through the redacted-OTLP → Alloy → Loki path. Lines carry deltas, so sum_over_time in LogQL yields exact request counts over any range. Idle nodes log nothing.
• OTel counter for the future metrics backend — dkg.chain.rpc.requests.total{rpc_method (bounded, else 'other'), chain_id}.
• Grafana — fleet: RPC requests per node + RPC requests by method (fleet); per-node: RPC requests by method — $node. Plus a ready-made RPC-spike alert example (would have tripped within the first hour of the incident).

Correctness evidence

• Real-path test: adapter over a live loopback JSON-RPC server → counter emitted with exactly {rpc_method, chain_id} labels; drain returns the delta and resets; lifetime monotonic.
• Bounded label mapping (unknown method → other); tracker never throws.
• cli formatter contract pinned (logfmt shape, delta-only, token sanitization, idle → no lines).
• Full chain suite 758 green — the counting provider is transparent to all failover behavior. Full monorepo build green.

Rollout

Rides the same node rollout as the log collection (needs a build with this + telemetry.logs.exporter: "otlp"). Panels light up as soon as nodes ship logs.

🤖 Generated with Claude Code

[Jurij89]

Adversarial review summary (RPC accounting / logging / dashboards)

I reviewed the counting provider, retry hook, adapter drain boundary, daemon emission, tests, dashboard queries, and example alert. I did not find a blocker in the core accounting design: first attempts are counted in _send, ethers retry attempts are counted through the retry hook, batchMaxCount: 1 keeps the count aligned with HTTP requests, idle windows stay quiet, and daemon shutdown does a final best-effort drain.

Non-blocking findings:

• [P3] The log path has looser method-cardinality bounds than the metric path. RpcUsageTracker.record() stores raw method names in the window while only the OTel metric label is bounded (packages/chain/src/rpc-usage.ts:120-127), and formatRpcUsageLines() emits any token-safe method into rpc_usage method=... (packages/cli/src/daemon/rpc-usage-log.ts:38-40). Grafana then groups logs by method. Today the adapter mostly generates known ethers methods, but future/custom calls could create noisy Loki query-time series. I would bound the log method label with the same allowlist used for metrics, or intentionally document the log as raw/high-cardinality.

• [P3] Multi-endpoint failover accounting is not directly covered by tests. The adapter switches to perEndpointRetries = 0 for multi-RPC nodes and wires counting through the shared provider factory (packages/chain/src/evm-adapter-base.ts:630-644), while retry attempts are counted in the retry hook (packages/chain/src/evm-adapter-rpc.ts:108-114). Existing tests cover healthy single-endpoint and single-endpoint 429 retries; I would add a multi-endpoint failover case to prove failed endpoint attempts and successful fallback attempts are both reflected in the drained window.

• [P3] The example RPC spike alert groups only by service_instance_id (tools/log-collection-poc/production/example-alerts.md:40). The fleet dashboard scopes by deployment environment, so reused node names across environments could merge credit-burn totals in the example alert. Consider grouping/filtering by deployment_environment and possibly chain as well.

Overall this looks production-usable from an observability standpoint once those hardening items are considered.

[Bojan131]

Thanks for the adversarial pass @Jurij89 — all three P3s addressed. What went where:

1. Log-path method cardinality → addressed in 875936418, with one deliberate deviation from the allowlist option: the log path stays raw by design, because unlike the metric label, method in the logs is parsed at Loki query time (| logfmt) — raw names cost no index cardinality, and full fidelity is the diagnostic point (when a node suddenly burns credits on debug_traceTransaction, we want the real name in the panel, not other). That rationale is now documented at the tracker. Your underlying concern (unbounded fabricated names → one rpc_usage line per name per minute) is real though, so the window now caps distinct raw keys at MAX_WINDOW_METHODS = 64 — overflow aggregates into other, existing keys keep counting raw. Unit-tested.

2. Multi-endpoint failover accounting → covered in 875936418: endpoint A rate-limits (HTTP 429 — a billed refusal, the request reached the provider), endpoint B serves; with the multi-RPC perEndpointRetries = 0 path the drained window must equal A hits + B hits, total and per method, with both servers asserted ≥1 (so the failed attempt and the fallback success are each proven billed). The test log confirms the real failover path runs (RPC failover: … failed (THROTTLE_429), trying …).

3. Example alert grouping by environment → correct catch, fixed in both places:

• This PR no longer carries that alert at all: the polaris-era dashboard/alert docs under tools/log-collection-poc/production/ were superseded by docs(observability): sync Grafana dashboards + alert docs to the production observability server #1414 (which moved the production observability config to tools/observability/ with Loki-3 query shapes and the burn alert in range+reduce form), so I dropped them here — that also removes a guaranteed merge conflict between the two PRs.
• The substantive fix landed in docs(observability): sync Grafana dashboards + alert docs to the production observability server #1414's alert catalog (aad7e5e64): all four per-node log rules (node-silent / error / warn / rpc-burn) now group by (service_instance_id, deployment_environment), summaries name the environment, and the notification routing group_by includes it — same-named nodes in different environments alert and notify independently. On "possibly chain as well": skipped deliberately — a node serves exactly one chain, so per-(node, environment) grouping already implies the chain; the chain= field stays on every line for filtering.

Verification: chain 6/6 (incl. the two new tests), cli 10/10, agent 4/4 locally; CI running on the new head.