Skip to content

Duration histograms reach 30 minutes; parked and canceled resume outcomes are named - #2642

Open
Davanum Srinivas (dims) wants to merge 2 commits into
agent-substrate:mainfrom
dims:metrics-buckets
Open

Davanum Srinivas (dims) wants to merge 2 commits into
agent-substrate:mainfrom
dims:metrics-buckets

Conversation

@dims

@dims Davanum Srinivas (dims) commented Oct 14, 2026 •

Copy link
Copy Markdown
Collaborator

Extends duration histogram buckets to 30 minutes so long request waits and snapshot operations can be measured beyond the previous 30–60 second limits. Router resume metrics now distinguish exhausted parking budgets (parked_exhausted), a full parking lot (parked_rejected), and canceled requests (canceled) instead of grouping them under unknown.

  • Tests pass: go test -race on atelet, extproc, ingress and ateattr; go vet; hack/verify/metrics.sh.
  • Appropriate changes to documentation are included in the PR: the metrics registry, docs/observability.md, docs/request-parking.md and docs/dev/best-practices/metrics.md.

@dims

Davanum Srinivas (dims) commented Oct 14, 2026 •

Copy link
Copy Markdown
Collaborator Author

trying to re-trigger the tests to overcome the flake

@dims Davanum Srinivas (dims) changed the title [WIP] Duration histograms reach 30 minutes; parked and canceled resume outcomes are named Duration histograms reach 30 minutes; parked and canceled resume outcomes are named Oct 14, 2026
@dims
Davanum Srinivas (dims) marked this pull request as ready for review October 14, 2026 14:13
@bowei Bowei Du (bowei) added area/network area/node area/observability kind/feature An enhancement / feature request or implementation labels Oct 14, 2026
@dims
Davanum Srinivas (dims) force-pushed the metrics-buckets branch 3 times, most recently from 72bc77f to 6e2d625 Compare October 17, 2026 01:48
Route, parking-wait, lifecycle and snapshot-phase histograms stopped at
30 s or 60 s, so parked waits and large restores all landed in +Inf.
ate.router.resume put budget exhaustion, lot rejection, client
cancellation and real failures all under unknown. Add parked_exhausted,
parked_rejected and canceled; unknown now means a failed resume or a
direction that resumes nothing.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area/network area/node area/observability kind/feature An enhancement / feature request or implementation

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants