[Bug]: NVSwitch health events reach no Prometheus metric: STORE_ONLY hardcoded and dcgm_health_active_events never populated

[lfriedman-netllama]

Description

NVSwitch health events reach the datastore and no Prometheus metric, by two independent routes. #1825 fixed the collection side; this is about what happens to the events afterwards.

Up front, the limitation in this report: I cannot demonstrate the runtime symptom. Our fleet is GB200 NVL72, where dcgmi discovery -l reports 0 NvSwitches found on every compute node, so the NVSwitch branch never executes for us and we have zero NVSwitch events to observe. Everything below is from reading the code on main, and the inference about runtime behaviour is labelled as such. Someone on an HGX or DGX baseboard with on-board NVSwitch can confirm it in a minute; I cannot.

1. STORE_ONLY is hardcoded at all three NVSwitch emit sites

health-monitors/gpu-health-monitor/gpu_health_monitor/platform_connector/platform_connector.py, lines 632, 658 and 683, each paired with componentClass="NVSWITCH" at 622, 647 and 676:

processingStrategy=platformconnector_pb2.STORE_ONLY,

commented "NVSwitch remediation is deferred until downstream handling is safe".

STORE_ONLY events are filtered before the Prometheus connector counts them, so they never appear in health_events_total. That much is a deliberate choice about remediation, and I am not arguing with deferring remediation. The side effect is that it also removes them from the counter that alerting is built on, which is a different concern from whether anything should act on them.

Comparable checks expose this as a value. gpuPowerBrakeStoreOnly lets an operator choose observe-only versus acted-on for the power brake. The NVSwitch path has no equivalent knob.

2. The NVSwitch branch never populates dcgm_health_active_events

This is the one I would fix first, because it looks like an oversight rather than a decision, and because it changes no remediation semantics at all.

That gauge is written from pending_metric_updates, drained at line 699:

for event_type, gpu_id, error_code, value in pending_metric_updates:
    metrics.dcgm_health_active_events.labels(...).set(value)

Every append to that list is in the GPU branch, at lines 501, 535 and 588. The NVSwitch branch (roughly 600 to 690) appends to health_events and to pending_cache_updates, and never to pending_metric_updates.

So the one metric that survives STORE_ONLY has no NVSwitch series either.

Combined effect

Inference, not observation: on a system where DCGM does enumerate switches, a DCGM_FR_NVSWITCH_FATAL_ERROR is detected, cached, and written to MongoDB, and is then invisible to health_events_total (filtered as STORE_ONLY) and to dcgm_health_active_events (never recorded). No alert is possible from either metric, and an operator watching a dashboard sees nothing.

That the fault is correctly attributed after #1825 makes this more noticeable, not less: the data is now right and still cannot be seen.

Suggested fixes, in the order I would do them

1. Append to pending_metric_updates in the NVSwitch branch, mirroring the GPU branch. Two notes from having made the same class of change in feat: label dcgm_health_active_events with the error code #1874: the clear path has to zero the same label set that was set, or a series latches at 1 for the process lifetime; and since feat: label dcgm_health_active_events with the error code #1874 the gauge carries error_code, so a check latched by several codes needs each one cleared.
2. Make the NVSwitch STORE_ONLY configurable, in the shape of gpuPowerBrakeStoreOnly. Detection-only deployments want these events counted while remediation stays off. I have lower confidence this is wanted, given the comment reads as deliberate, which is why it is second and separable.

Happy to write (1) if it is welcome, with the caveat that I cannot test it on our hardware and would be relying on CI plus a reviewer with switches.

Not duplicates, for the record

• [Bug]: GPU health monitor drops or misattributes DCGM NVSwitch incidents #1824 / fix(health-monitor): fix missing NVSwitch incidents report #1825 are the collection bug and its fix. This is about exposure after collection, and only became worth reporting once collection was correct.
• [Feature/Enhancement]: Fabric health signals for GB*/NVL72 rack-level architectures #1275 is fabric health on GB*/NVL72 where there is no host fabric manager and authority sits with rack-level controllers. That is a different problem on different hardware, and it is the one that actually affects my fleet. Not this.
• [Bug]: NVSentinel not detecting fabric health on H100s #883 is H100 fabric-manager process health, also distinct.

Environment

NVSentinel v1.24.0, chart v1.24.0, MongoDB datastore, detection-only. Line numbers verified against origin/main at the time of filing and are identical to the v1.24.0 tag.

[lalitadithya]

ah, this is a good catch, I can help test, please feel free to open up a PR if you are up for it

[lfriedman-netllama]

Opened #1910 for the metric half of this, since you offered.

It adds a dedicated dcgm_health_active_switch_events{event_type, switch_id, error_code} gauge and routes the NVSwitch failure, per-code recovery and full-recovery updates to it. A separate gauge rather than reusing dcgm_health_active_events because switch ids and GPU ids are distinct sequences that both start at 0, so sharing gpu_id would make gpu_id="3" match GPU 3 and switch 3 at once. CodeRabbit caught that on the first revision and it was a fair catch.

CI is green and it is waiting on review.

That leaves the other half of this issue, the hardcoded STORE_ONLY, untouched. Happy to take that too, but it is a behaviour change rather than an observability fix so I would rather agree the shape with you first.

Worth flagging for context: on our NVL72 fleet neither half is observable yet, for a reason upstream of both. DCGM enumerates zero NVSwitches on these nodes, so switch_ids comes back empty and the entityType="NVSWITCH" branch never runs at all. What does happen is that the NVSwitch watches evaluate the node's four local GPUs instead and report PASS:

"dcgm gpu_id are [0, 1, 2, 3]"
"Updated cache for key GpuNvswitchFatalWatch|GPU|0 ..."

Note the entity type in that cache key is GPU, not NVSWITCH. So today these checks cannot fail on NVL72, and they look healthy rather than looking absent, which is the worse of the two failure modes. #1275 covers the underlying switch-tray visibility gap. #1910 is still worth having, because it is the half that becomes correct the moment the enumeration does.