Add sustained-download detection; drop dead stick-table peer listener - #79
Merged
Conversation
Remove the HAProxy stick-table peer listener. It was dead code: NewServer was
referenced only from its own e2e test, Config.HAProxyPort was assigned and
never read, and examples/haproxy.cfg has no peers section. README advertised a
feature that never ran.
Replace it with detection for a class of abuse rate limits cannot see. A
scraper that stays under every per-second limit but sustains that rate for an
hour across thousands of resources is invisible to a rate threshold and obvious
over a five-minute window.
Two independent selection paths over the same sliding window:
volume - many requests, many resources, many sections, many bytes.
shape - breadth with no byte floor at all, gated by a resources-per-section
ceiling. Enumeration is defined by breadth, so requiring bytes would
mean only catching it once it also became a volume problem.
Breadth comes from the existing SPOE feed; host and path are already in
HTTPContext, so no HAProxy config change is needed. Bytes come from new eBPF TC
egress per-client counters, because HAProxy cannot report transferred bytes over
SPOE at all: its byte counters are stream-scoped, on-http-response fires before
the body moves, and a fresh stream is allocated per keep-alive request. The
egress counters see real wire bytes including streamed and chunked responses.
Hostnames and paths are hashed on the way in and never retained - the tracker
only counts and compares distinct values.
Reputation raises request and byte floors rather than exempting, and does not
raise breadth floors, since breadth is not an entitlement. Selected clients are
held, because blocking destroys the byte evidence that selected them; release is
judged on requests and breadth only, and a hard ceiling bounds the hold because
once enforcement removes the evidence nothing separates a false positive from a
suppressed true positive.
Detect-only by default, with -sustained-enforce a separate flag, and
/api/sustained reporting per client which thresholds it is under and how close
it is to each.
Also add an analyzer-wide runtime enforcement kill switch at the point commands
are issued (POST /api/enforcement/stop). It covers every detector, not one:
"we are blocking traffic we should not be" is not known up front to be confined
to a single detector, so an operator should not have to identify the culprit
before they can stop it. Relieving commands are not suppressed, since the switch
is pulled precisely when a block is wrong. One-way; resuming needs a restart.
Validation: gofmt, go vet, make portable-test, and targeted tests all pass on
macOS. The eBPF object was compiled with clang and loaded through a real kernel
verifier in Docker (all three programs, all 26 maps), but on a 7.0 kernel rather
than 5.4, and no live-traffic test was possible from this host. make collector,
make test, and the TC/XDP attach paths were not run.
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Production rollout surfaced two gaps. The collector serves metrics from an allowlist registry, so the egress accounting counters were incremented but never scrapeable. The analyzer registers the same names and reports a constant 0 for them, so there was no way to tell "accounting is off" from "nothing crossed the floor" -- exactly the distinction the two counters exist to make. Register them on the collector's registry and say in the docs which process to scrape. Removing -haproxy-port is a breaking upgrade: Go's flag parser rejects unknown flags, so a collector still passing it exits with status 2, and Restart=on-failure turns that into a crash loop. A systemd drop-in that overrides ExecStart= shadows the main unit, so editing the unit alone is not enough -- document `systemctl cat` as the check that catches it. Validated on production hosts (kernel 6.8, live traffic): eBPF egress maps populate per client, signals reach the analyzer and are attributed to the right IP, the byte floor gates as designed, and the analyzer-wide kill switch flips /api/enforcement and packetyeeter_enforcement_stopped. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
The deb replaces the unit file, so removing -haproxy-port from the shipped ExecStart= is enough for a stock install. It is not enough when a local drop-in overrides ExecStart=: the drop-in shadows the unit and the upgrade leaves it untouched, so the flag is still passed. Go's flag parser rejects unknown flags, so the collector exits 2 and Restart=on-failure turns that into a crash loop. Because postinstall deliberately does not restart the service, the breakage surfaces at the next restart, detached from the upgrade that caused it. Check the resolved configuration in postinstall and name the offending files. The package cannot safely edit operator-authored drop-ins, so this warns rather than modifies, and does not fail the install. Verified on a host carrying such a drop-in: silent when the resolved config is clean, names the drop-in when it is not.
Stray `go build -o analyzer ./cmd/analyzer` output at the repo root is not covered by the existing ignore rules, which only match the packetyeeter-* names. Add the bare names anchored to the repo root -- unanchored `analyzer` and `collector` would also match the pkg/ and cmd/ source directories. Verified that no tracked file becomes ignored. Switch the docs to example.com hostnames and a documented $COLLECTOR_HOST variable so the commands are copy-pasteable against any deployment.
awlx
force-pushed
the
awlx-sustained-download-detection
branch
from
August 9, 2026 07:47
4cc4a95 to
fe546a9
Compare
Records the new detector, the analyzer-wide enforcement kill switch, the removal of the dead HAProxy peer listener, and the breaking -haproxy-port removal with the drop-in check operators need before upgrading.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds detection for clients that pull large volumes over a long window — the "slow, patient scraper" shape that rate-based detection misses because no individual interval looks abusive. Selection is on duration and breadth, not rate.
Two independent signals, AND-ed:
bytes_outis per-stream and zeroed bystream_new, and keep-alive allocates a new stream per request, so it never accumulates).res.body_sizeonly gives advertisedContent-Length, useless for streamed responses — deliberately not used as a fallback rather than introducing a second, inconsistent byte source.Also removes the HAProxy peer (stick-table) listener. It was dead code: it blocked on any stick-table update with no volume logic and was never wired to anything.
Enforcement kill switch
POST /api/enforcement/stophalts every enforcing command analyzer-wide while detection keeps running. Deliberately placed at the command-issuing boundary (Broadcast+sendCommand) rather than inside one detector, so it covers all of them. It is checked before the dedup reservation — otherwise a suppressed block would mark the IP "recently blocked" and stay suppressed for the dedup TTL after enforcement resumed. One-way by design: resuming needs a config change and a restart.Safety
Everything ships off by default, and enabling detection does not enable blocking (
-sustained-enabledvs-sustained-enforce). Design choices that bound false-positive cost:("ab","c") != ("a","bc").-haproxy-portis removed. Go's flag parser rejects unknown flags, so a collector still passing it exits 2, andRestart=on-failureturns that into a crash loop. A local systemd drop-in overridingExecStart=shadows the shipped unit and survives the upgrade — this is a real failure that happened during testing.collector-postinstall.shnow inspects the resolved config and names the offending files; it warns rather than edits, since the package has no business rewriting operator-authored drop-ins.Check before upgrading:
Housekeeping
Adds root-anchored ignore rules so stray
go build -o analyzeroutput at the repo root can't be committed, and switches the docs to placeholder hostnames.Validation
gofmt,go vet,git diff --check,make portable-test(13/13 packages) all clean.egress_bytesmaps populated per client, signals reached the analyzer attributed to the correct IP, the byte floor gated as designed (796 signals at a 4 KiB test floor vs 1 at the production 1 MiB floor),/api/sustainedtracked hundreds of clients with blockers/margins populated, and the kill switch flipped both the endpoint andpacketyeeter_enforcement_stopped.SIGNAL_EGRESS_VOLUMEwithout erroring, so a collector-first rollout is safe.Live testing also caught a bug not visible in unit tests: the collector serves metrics from an allowlist registry, so the egress counters were incremented but never scrapeable. They're now registered on the collector's registry, and the docs say which process to scrape (the analyzer registers the same names and reports a constant 0).
A code review during development caught one high-severity bug, fixed here with a regression test verified to fail against the old code: the IPv6 map iterator reuses one
[16]bytekey per poll, and signals are marshaled asynchronously off the queue, so storing the caller'snet.IPdirectly meant a queued signal could report the wrong client.