Fix DeletePartialMatch deleting unrelated metrics on hash collision - #2081
Fix DeletePartialMatch deleting unrelated metrics on hash collision#2081pujitha24 wants to merge 1 commit into
Conversation
Motivation: Issue prometheus#1810 reports metrics not being freed as expected after calling DeletePartialMatch(). While investigating, a distinct, reproducible bug was found in metricMap.deleteByLabels (the function backing MetricVec.DeletePartialMatch): when two or more differently-labeled metrics land in the same hash bucket (a collision), and only one of them matches the partial-label deletion criteria, the old code deleted the *entire* bucket via `delete(m.metrics, h)` instead of only the matching entry - silently discarding unrelated metrics that were never meant to be deleted. Delete() and DeleteLabelValues() already handled this correctly elsewhere in the same file (splicing out only the matching slice element); DeletePartialMatch did not. Additionally, the old single-match code path did not stop after removing one bucket, and any collision-safe removal needs to clear the vacated slice slot so it does not keep referencing a Metric object through the backing array's spare capacity, which would prevent that Metric from being garbage collected. This bug is a plausible contributor to reports like prometheus#1810, but the original reporter's case was never profiled, so it is not confirmed as the root cause. Hash collisions on the label-value hash are rare in practice, so this alone may not fully explain a leak; regardless, the incorrect bucket-wide deletion is a real correctness bug worth fixing on its own. Approach: Rewrite deleteByLabels to filter each hash bucket's slice in place, removing only the entries that actually match the partial-label criteria (there can be more than one match per bucket), and explicitly zero the vacated tail slots of the reused backing array so deleted Metric objects don't stay reachable. Removed findMetricWithPartialLabels, which became unused once its only call site was replaced. Validation: Added TestDeletePartialMatchWithCollisions, following the existing TestDeleteWithCollisions / TestDeleteLabelValuesWithCollisions pattern: forces all label combinations into a single hash bucket via a stubbed-out hashAdd/hashAddByte, then reuses the existing testDeletePartialMatch test body. Confirmed this test fails against the pre-fix code (unrelated collided metrics vanish) and passes after the fix. go build ./... go test ./prometheus/... -run 'TestDelete|TestMetricVec|TestCurryVec' -race -v All of the above passed. (The full `go test ./prometheus/...` has two pre-existing, unrelated failures on this checkout's Go toolchain, caused by a godebug runtime metric mismatch in go_collector_latest_test.go; they reproduce identically on main without this change and are unaffected by it.) Fixes prometheus#1810 Signed-off-by: Pujitha Paladugu <10557236+pujitha24@users.noreply.github.com>
|
Just checking in on this — it's still rebased and green, happy to make any changes that would help move review along. |
|
Again, hash collision is not really a common occurence, unless you we can prove otherwise Can you provide a repro test that confirms that e.g. constructs two series that collide? |
|
|
|
recently, when I use AI review code, it report a bug about this same issue: |
|
Thanks for double-checking with an AI reviewer. The pattern it describes is exactly the bug this PR fixes — that write-up matches the old, pre-fix |
That's my exact point. If it's not feasible, it means it's unlikely to happen on production, then why fixing? |
Motivation:
Issue #1810 reports metrics not being freed as expected after calling
DeletePartialMatch(). While investigating, a distinct, reproducible bug
was found in metricMap.deleteByLabels (the function backing
MetricVec.DeletePartialMatch): when two or more differently-labeled
metrics land in the same hash bucket (a collision), and only one of
them matches the partial-label deletion criteria, the old code deleted
the entire bucket via
delete(m.metrics, h)instead of only thematching entry - silently discarding unrelated metrics that were never
meant to be deleted. Delete() and DeleteLabelValues() already handled
this correctly elsewhere in the same file (splicing out only the
matching slice element); DeletePartialMatch did not.
Additionally, the old single-match code path did not stop after
removing one bucket, and any collision-safe removal needs to clear the
vacated slice slot so it does not keep referencing a Metric object
through the backing array's spare capacity, which would prevent that
Metric from being garbage collected.
This bug is a plausible contributor to reports like #1810, but the
original reporter's case was never profiled, so it is not confirmed as
the root cause. Hash collisions on the label-value hash are rare in
practice, so this alone may not fully explain a leak; regardless, the
incorrect bucket-wide deletion is a real correctness bug worth fixing
on its own.
Approach:
Rewrite deleteByLabels to filter each hash bucket's slice in place,
removing only the entries that actually match the partial-label
criteria (there can be more than one match per bucket), and explicitly
zero the vacated tail slots of the reused backing array so deleted
Metric objects don't stay reachable. Removed findMetricWithPartialLabels,
which became unused once its only call site was replaced.
Validation:
Added TestDeletePartialMatchWithCollisions, following the existing
TestDeleteWithCollisions / TestDeleteLabelValuesWithCollisions pattern:
forces all label combinations into a single hash bucket via a
stubbed-out hashAdd/hashAddByte, then reuses the existing
testDeletePartialMatch test body. Confirmed this test fails against the
pre-fix code (unrelated collided metrics vanish) and passes after the
fix.
go build ./...
go test ./prometheus/... -run 'TestDelete|TestMetricVec|TestCurryVec' -race -v
All of the above passed. (The full
go test ./prometheus/...has twopre-existing, unrelated failures on this checkout's Go toolchain, caused
by a godebug runtime metric mismatch in go_collector_latest_test.go;
they reproduce identically on main without this change and are
unaffected by it.)
Fixes #1810
Signed-off-by: Pujitha Paladugu 10557236+pujitha24@users.noreply.github.com