Sync Grafana dashboards with pkg/metrics - #80
Merged
Conversation
Three panel targets queried measurements that no longer exist and rendered empty: packetyeeter_baseline_anomalies (now _total), packetyeeter_baseline_stats (now packetyeeter_baseline_calibrated_asns) and packetyeeter_ml_stats (now packetyeeter_ml_predictions_total). 57 registered metrics were not graphed anywhere. That included the whole sustained-download subsystem, the enforcement kill-switch and every queue depth/drop gauge - the series needed during a staged rollout to tell "detection is quiet" apart from "detection is not running". Added rows for sustained download, enforcement safety, pipeline backpressure, attack campaigns and carpet bombing, clock skew and payload entropy, ML/AI engine health, and protocol/SPOE/reputation. Both dashboards now cover every metric registered in pkg/metrics plus the reputation gauges. Privacy is unchanged. Panels backed by per-IP, per-JA4H or per-user-agent series aggregate those labels away: the per-IP detection counter is summed inside a subquery so the address never reaches the panel, and threat_intel_info and ai_recent_detections are reduced to series counts. All of these are gated behind -enable-high-cardinality-metrics and stay empty without it. Also repacked gridPos. The main dashboard had 18 overlapping panel rectangles and the overview had 2, which Grafana silently reflowed on import, so the checked-in layout did not match what operators saw. Panels now carry unique ids. The files stay in the classic schema V1 format. Schema V2 is experimental in Grafana 12.0-12.2, is gated behind dashboardNewLayouts, and V2 dashboards cannot be viewed at all without that toggle, so shipping V2 would break import for most users. Documented the resulting "Annotations is not an array" import error in docs/observability.md.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
The checked-in dashboards had drifted from
pkg/metrics.Three panel targets queried measurements that no longer exist and rendered as empty panels:
packetyeeter_baseline_anomaliespacketyeeter_baseline_anomalies_totalpacketyeeter_baseline_stats.calibrated_asnspacketyeeter_baseline_calibrated_asnspacketyeeter_ml_stats.sample_countpacketyeeter_ml_predictions_total57 of the registered metrics were not graphed anywhere. That included the entire sustained-download subsystem (#79), the enforcement kill-switch, and every queue depth/drop gauge — exactly the series an operator needs during a staged rollout to tell "detection is quiet" apart from "detection is not running".
Changes
Fixed the three stale targets and added rows covering the rest:
Both dashboards now cover every metric registered in
pkg/metricsplus thepkg/analyzer/reputationgauges.grafana-dashboard.jsongoes 77 → 123 panels,grafana-dashboard.influx.json28 → 38. No pre-existing panel or query was dropped (verified againstHEAD; the only three removed targets are the stale ones above).Privacy is unchanged
Panels backed by per-IP, per-JA4H, or per-user-agent series aggregate those labels away, so the shared dashboards still never render a client address:
packetyeeter_ai_detections_by_ip_totalis summed inside a subquery, so the derivative is still taken per-series but theiptag never reaches the panel.packetyeeter_threat_intel_infoandpacketyeeter_ai_recent_detectionsare reduced to series counts.packetyeeter_clock_skew_ppmandpacketyeeter_payload_entropy_bitsrender as aggregate averages.packetyeeter_ja4db_user_agent_hits_totalgroups by fingerprint/match type only, never the UA string.packetyeeter_reputation_scoreis averaged per entity type, never per key.All of these are gated behind
-enable-high-cardinality-metricsupstream and stay empty without it; panel descriptions say so.Layout
Repacked
gridPosin both files. The main dashboard had 18 overlapping panel rectangles and the overview had 2, which Grafana silently reflowed on import — so the checked-in layout did not match what operators actually saw. Panels now also carry unique ids (48 previously had none).Schema version
The files stay in the classic schema V1 format. Schema V2 was considered and rejected: it is experimental in Grafana 12.0–12.2, gated behind
dashboardNewLayouts, and V2 dashboards cannot be viewed at all without that toggle, so shipping V2 would break import for most users. Importing V1 into an instance that does have the toggle on fails withAnnotations is not an array; that's now documented indocs/observability.mdalong with the fix (import with the toggle off).Also updated
docs/observability.md, which previously stated the campaign panels were intentionally left out of the dashboard JSON.Validation
make portable-test— passpkg/metricsvs both dashboards: 0 stale refs, 0 uncovered metricsgit diff --checkcleanNo Go code changed, so no Linux/eBPF validation was required. Dashboard-only change; nothing here alters detection or enforcement behavior.