Skip to content

collector/topology: label/taint disambiguation map keys can collide with real dotted label names, silently dropping readings #2003

Description

@yuanchen8911

Problem

encodeLabels (pkg/collector/topology/topology.go) disambiguates a label key carrying multiple distinct values as <key>.<value>. That synthesized map key can collide with a different label literally named <key>.<value>: both write to the same entry in the output map, and one silently overwrites the other, nondeterministically by Go map iteration order.

Concrete case (reproduced): real label k is true on gpu-a and false on gpu-b (→ disambiguated entries k.true, k.false), while a distinct label named k.true with value true exists on both nodes (→ plain entry keyed k.true). The two k.true writes race; either a genuine reading or the distinct label's reading is lost from the snapshot.

This is silent data loss in the collector output, independent of any consumer. encodeTaints has the same shape (<key>.<effect> disambiguation).

Downstream impact

The node-set constraint form (#1755, PR #2000) fails closed on the detectable signature (overlapping node sets across an accepted disambiguated set violate the one-value-per-node partition), but detection-side heuristics cannot recover the lost reading, and collisions with identical node sets are undetectable in principle.

Proposed fix

Make the encoding lossless: separate the key and value structurally instead of joining them with . — e.g. reserve a separator that cannot appear in label keys, or move label readings to Subtype.Items with explicit key/value/nodes fields (a snapshot-format change, so version-gated). Related: #2002 tracks structured truncation metadata; a combined structured encoding could resolve both.

Context

Found during Codex review of PR #2000; the evaluator-side fail-closed guard shipped there, this issue tracks the collector-side root cause.

Metadata

Metadata

Assignees

No one assigned

    Type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions