Rate limiting observability (metrics and tracing)

[jerm-dro]

User Story

As an SRE,
I want metrics and tracing for rate limiting,
so that I can monitor enforcement and troubleshoot issues in production.

Context

See THV-0057: Rate Limiting for MCP Servers for full design details.

Acceptance Criteria

• Decision counter increments on allowed and rejected requests, broken down by scope and operation type
• Redis error counter increments on Redis failures, broken down by error type
• Fail-open counter increments when a request is allowed through during Redis outage
• Check latency histogram records Lua script round-trip time
• All metrics use toolhive_ prefix following project conventions
• Span attributes rate_limit.decision, rate_limit.rejected_by, rate_limit.fail_open present on request span
• Unit: Verify each metric increments under the correct conditions
• Unit: Verify span attributes are set on allowed, rejected, and fail-open paths
• E2E: Send rate-limited traffic, scrape metrics endpoint, verify counters are non-zero

Dependencies

• STORY-001 (core rate limit middleware exists to instrument)

Out of Scope

• Alert definitions and runbooks (operational docs)
• Dashboard templates

[Sanskarzz]

@jerm-dro I would like to work on this issue please assign.

[jerm-dro]

@Sanskarzz done!

[Sanskarzz]

@jerm-dro For the rate-limit decision counter, should we increment once per request, or once per evaluated rate-limit scope?

For example, if one request checks shared, per-user, and per-tool buckets, should that produce one metric sample or three? I recommend once per evaluated scope because it gives better production visibility into which limits are enforcing traffic.

[jerm-dro]

Finished by #5701