coord/cluster: latency-gradient tracker minRTT/curRTT (v0.12 phase 3) - #4
Merged
Conversation
Adds a loop-owned, deterministic RTT gradient tracker to the coordinator
(ADR-0039 part 2), one per operation type since a PUT (k+m shards) and a GET (k
shards) have different cost profiles. Per op it maintains minRTT — the best-case
service time, a re-probing two-window minimum that can RISE again when the
genuine floor degrades (the signal part 5 watches) — and curRTT, a short-window
EWMA of recent latency; the gradient is clamp(minRTT/curRTT, 0..1), ~1 healthy
and ->0 as queuing inflates curRTT above the floor, reading 1 (healthy) before a
short warmup so a fresh node never looks overloaded.
It is fed from the same per-op sample observeLatency reports — timed once on the
seam clock, handed to both the histogram and the tracker — so the two never
diverge. Exposed as hamster_s3_request_{gradient,min_rtt_seconds,cur_rtt_seconds}
{method} via the coordinator-accessor + scrape-collector pattern the durability
gauges use, so the coordinator still never imports internal/metrics. Names share
the hamster_s3_request_ prefix with the duration histogram so the request-level
signals group together.
Computes and exposes the signal only; shedding comes in part 3/4.
Signed-off-by: Nicholas Phillips <nsphilli@gmail.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
v0.12 adaptive load shedding — Phase 3 of 5 (ADR-0039 part 2), on top of Phase 2's per-op latency.
Adds a loop-owned, deterministic RTT gradient tracker to the coordinator, one per op type (PUT/GET — different cost profiles):
minRTT— best-case service time, a re-probing two-window minimum that can rise when the genuine floor degrades (the signal Phase 5 watches; not a permanent all-time min).curRTT— short-window EWMA of recent latency.gradient = clamp(minRTT/curRTT, 0..1)— ≈1 healthy, →0 as queuing inflatescurRTT; reads 1 (healthy) before a 5-sample warmup so a fresh node never looks overloaded.Design:
observeLatencytimes once on the seam clock and feeds both the Phase-2 histogram and the tracker, so they can never diverge.Gradient/MinRTT/CurRTTaccessors;internal/clusterreads them on the loop via a scrape collector — the coordinator still never importsinternal/metrics.hamster_s3_request_{gradient,min_rtt_seconds,cur_rtt_seconds}{method}— I aligned the prefix with the Phase-2hamster_s3_request_duration_secondsso all four request-level signals group together.Constants (ADR leaves these to implementation):
minWindowSamples=200,curAlpha=0.2,gradientWarmup=5— all commented.Tests: pure unit (
TestGradientWarmup,TestGradientFallsAsLatencyRises,TestMinRTTRisesAfterReprobe,TestEmptyTracker,TestCoordinatorAccessors) + sim (TestGradientTracksObservedLatency, assertingminRTT == min(observed)→ single source).task check/build/test+-raceon the tracker green. (One unrelated load-induced flake in the real-process cluster suite passed on re-run — CI's cluster job has the bounded retry for it.)