Skip to content

Sync Grafana dashboards with pkg/metrics - #80

Merged
awlx merged 1 commit into
mainfrom
awlx-grafana-dashboard-metric-coverage
Aug 9, 2026
Merged

Sync Grafana dashboards with pkg/metrics#80
awlx merged 1 commit into
mainfrom
awlx-grafana-dashboard-metric-coverage

Conversation

@awlx

@awlx awlx commented Aug 9, 2026

Copy link
Copy Markdown
Owner

Problem

The checked-in dashboards had drifted from pkg/metrics.

Three panel targets queried measurements that no longer exist and rendered as empty panels:

Stale query Actual metric
packetyeeter_baseline_anomalies packetyeeter_baseline_anomalies_total
packetyeeter_baseline_stats.calibrated_asns packetyeeter_baseline_calibrated_asns
packetyeeter_ml_stats.sample_count packetyeeter_ml_predictions_total

57 of the registered metrics were not graphed anywhere. That included the entire sustained-download subsystem (#79), the enforcement kill-switch, and every queue depth/drop gauge — exactly the series an operator needs during a staged rollout to tell "detection is quiet" apart from "detection is not running".

Changes

Fixed the three stale targets and added rows covering the rest:

  • Sustained Download Detection
  • Enforcement Safety
  • Pipeline Backpressure
  • Attack Campaigns & Carpet Bombing
  • Clock Skew & Payload Entropy
  • ML & AI Engine Health
  • Protocol Rates, SPOE & Reputation

Both dashboards now cover every metric registered in pkg/metrics plus the pkg/analyzer/reputation gauges. grafana-dashboard.json goes 77 → 123 panels, grafana-dashboard.influx.json 28 → 38. No pre-existing panel or query was dropped (verified against HEAD; the only three removed targets are the stale ones above).

Privacy is unchanged

Panels backed by per-IP, per-JA4H, or per-user-agent series aggregate those labels away, so the shared dashboards still never render a client address:

  • packetyeeter_ai_detections_by_ip_total is summed inside a subquery, so the derivative is still taken per-series but the ip tag never reaches the panel.
  • packetyeeter_threat_intel_info and packetyeeter_ai_recent_detections are reduced to series counts.
  • packetyeeter_clock_skew_ppm and packetyeeter_payload_entropy_bits render as aggregate averages.
  • packetyeeter_ja4db_user_agent_hits_total groups by fingerprint/match type only, never the UA string.
  • packetyeeter_reputation_score is averaged per entity type, never per key.

All of these are gated behind -enable-high-cardinality-metrics upstream and stay empty without it; panel descriptions say so.

Layout

Repacked gridPos in both files. The main dashboard had 18 overlapping panel rectangles and the overview had 2, which Grafana silently reflowed on import — so the checked-in layout did not match what operators actually saw. Panels now also carry unique ids (48 previously had none).

Schema version

The files stay in the classic schema V1 format. Schema V2 was considered and rejected: it is experimental in Grafana 12.0–12.2, gated behind dashboardNewLayouts, and V2 dashboards cannot be viewed at all without that toggle, so shipping V2 would break import for most users. Importing V1 into an instance that does have the toggle on fails with Annotations is not an array; that's now documented in docs/observability.md along with the fix (import with the toggle off).

Also updated docs/observability.md, which previously stated the campaign panels were intentionally left out of the dashboard JSON.

Validation

  • make portable-test — pass
  • Coverage diff of pkg/metrics vs both dashboards: 0 stale refs, 0 uncovered metrics
  • JSON parses; 0 gridPos overlaps; unique panel ids; balanced quotes/parens in all 218 targets
  • git diff --check clean

No Go code changed, so no Linux/eBPF validation was required. Dashboard-only change; nothing here alters detection or enforcement behavior.

Three panel targets queried measurements that no longer exist and rendered
empty: packetyeeter_baseline_anomalies (now _total), packetyeeter_baseline_stats
(now packetyeeter_baseline_calibrated_asns) and packetyeeter_ml_stats (now
packetyeeter_ml_predictions_total).

57 registered metrics were not graphed anywhere. That included the whole
sustained-download subsystem, the enforcement kill-switch and every queue
depth/drop gauge - the series needed during a staged rollout to tell "detection
is quiet" apart from "detection is not running". Added rows for sustained
download, enforcement safety, pipeline backpressure, attack campaigns and carpet
bombing, clock skew and payload entropy, ML/AI engine health, and
protocol/SPOE/reputation. Both dashboards now cover every metric registered in
pkg/metrics plus the reputation gauges.

Privacy is unchanged. Panels backed by per-IP, per-JA4H or per-user-agent series
aggregate those labels away: the per-IP detection counter is summed inside a
subquery so the address never reaches the panel, and threat_intel_info and
ai_recent_detections are reduced to series counts. All of these are gated behind
-enable-high-cardinality-metrics and stay empty without it.

Also repacked gridPos. The main dashboard had 18 overlapping panel rectangles
and the overview had 2, which Grafana silently reflowed on import, so the
checked-in layout did not match what operators saw. Panels now carry unique ids.

The files stay in the classic schema V1 format. Schema V2 is experimental in
Grafana 12.0-12.2, is gated behind dashboardNewLayouts, and V2 dashboards cannot
be viewed at all without that toggle, so shipping V2 would break import for
most users. Documented the resulting "Annotations is not an array" import error
in docs/observability.md.
@awlx
awlx merged commit 9ef9bff into main Aug 9, 2026
4 checks passed
@awlx
awlx deleted the awlx-grafana-dashboard-metric-coverage branch August 9, 2026 09:29
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant