Fix production analyzer and collector reliability - #81
Merged
Conversation
…lock The reputation-threshold block path is gated on `a.MLModel != nil`, but `a.MLModel` was constructed unconditionally at startup regardless of `-ml-model`. `NewModelManager()` hardcodes `enabled: true` with an untrained `SimpleThresholdModel`, so deployments that never opted into ML still had every reputation block vetoed by it. That instance is never trained either -- the only `Train()` call sites target the separate AI-engine model -- so `sampleCount` stays below `minSamples` forever and prediction always comes from a pristine fallback. The statistical model also could not return `IsBot: true` for any input. `signalRateWeight` (0.15) and `diversityWeight` (0.10) were declared and initialized but never referenced in `Predict()`, stranding 25% of the weight mass and capping achievable confidence at 0.61 -- below the 0.65 `botThreshold`. The gate then compared against a hardcoded 0.7 rather than the operator's `-ai-confidence-threshold`, so even a repaired model would have been measured against the wrong bar. Observed across two production analyzers running without `-ml-model`: 853k `packetyeeter_ml_blocks_overridden_total`, zero confirmations, peak observed confidence 0.351, and ~50 warn-level lines per second from the two reject sites. - Construct `a.MLModel` only when `-ml-model` is set; otherwise the gate stays out of the decision path entirely. - Apply all eight declared weights in `Predict()`; they now sum to 1.0 and a maximally hostile entity scores 0.936 (was 0.61). - Read `ReputationScore` on its real scale and polarity in `calculateBehavioralScore`. The old `ReputationScore < 0.3` test read a raw, unbounded engine score (higher = worse) as a normalized 0-1 trust value, so the branch was dead for every entity the gate actually sees. - Use `-ai-confidence-threshold` instead of the hardcoded constant. - Drop the two per-signal rejection logs to debug; MLBlocksOverridden remains the aggregate signal. Operators running without `-ml-model` were effectively in monitor mode for reputation-based blocking regardless of `-dry-run`, so thresholds should be re-tuned before enforcement is enabled. Validated with `make portable-test` on macOS. Linux/eBPF collector paths (`make bpf`, `make collector`, `make test`) were not exercised; this change touches analyzer/ML userspace code only. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Re-evaluate the prior ML gate change and replace it with a safety-preserving implementation: - share one validated HybridModel between AI detection and the optional reputation gate - leave reputation blocking ungated when -ml-model is unset - fail startup when a configured ONNX model cannot load instead of silently falling back to an untrained statistical heuristic - synchronize ONNX close/reload, clean provisional sessions on load errors, drain RPCs before model destruction, and unwind startup resources - resolve and validate the AI confidence threshold once and use an inclusive boundary consistently - retain existing statistical scoring calibration, removing only the inverted pristine-reputation bonus - gate exact ASN/org metric families behind the existing high-cardinality flag - back off collector reconnects after unstable streams - expose kernel perf-ring lost samples in collector metrics Traffic enforcement policy is unchanged. UDP fragment handling, kernel rate limits, and reputation score caps remain untouched because changing those requires representative traffic analysis and operator policy decisions. Validated with make portable-test and targeted race tests on macOS. Linux/eBPF runtime validation will be performed on the production hosts in dry-run mode. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Initialize both bounded perf reader labels at collector metrics startup so operators can distinguish zero loss from a missing/unregistered metric before the first loss event occurs. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Keep wildcard and unclassified JA4DB matches as enrichment instead of high-severity bot signals. Preserve observe-only campaign semantics by removing reputation side effects, and report only detection-qualified active campaigns. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
If service_http omits the ja4_info on a wildcard match (to avoid emitting a high-severity signal for a coarse prefix), CategorizeBot would previously re-query the database. Since it used LookupWithType (which drops the MatchType), it would falsely treat the wildcard as an exact match and still use it to short-circuit crawler categorization, defeating the entire purpose of the prior fix. Updated JA4Verifier interface and CategorizeBot to require an exact match when querying the DB as a fallback.
IsKnownBot and GetInfo previously reused Lookup(), which includes coarse wildcard hits. That let LookupJA4H report IsWildcardMatch=false for a wildcard bot collision and inflated known-bot metrics/logs on HTTP path matches that were never allowed to emit signals. Require exact catalog hits for attribution helpers, format info from the existing LookupResult in the HTTP path, report wildcard honestly in the JA4H RPC, and add regression tests for CategorizeBot/IsKnownBot/GetInfo. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
…ights Default fragmented UDP / IPv6 fragments to rate-limit instead of hard-drop, edge-trigger kernel rate-limit incidents, expose analyzer reputation score caps with finite defaults, and calibrate statistical model weights so Predict mass sums to 1.0 without inventing labeled thresholds. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
HybridModelfor analyzer inference and reputation confirmation, and leave reputation ungated when-ml-modelis unsetMotivation
An unconfigured statistical model could veto reputation blocks, coarse JA4DB matches could be promoted to high-severity bot detections, and observe-only campaign findings could indirectly affect enforcement through reputation. Metrics also had avoidable cardinality and campaign-count accuracy problems. Collector reconnects could spin after short-lived streams, and perf-ring loss was not observable.
Validation