Skip to content

Add sustained-download detection; drop dead stick-table peer listener - #79

Merged
awlx merged 5 commits into
mainfrom
awlx-sustained-download-detection
Aug 9, 2026
Merged

Add sustained-download detection; drop dead stick-table peer listener#79
awlx merged 5 commits into
mainfrom
awlx-sustained-download-detection

Conversation

@awlx

@awlx awlx commented Aug 9, 2026

Copy link
Copy Markdown
Owner

Summary

Adds detection for clients that pull large volumes over a long window — the "slow, patient scraper" shape that rate-based detection misses because no individual interval looks abusive. Selection is on duration and breadth, not rate.

Two independent signals, AND-ed:

  • Volume — a new eBPF TC-egress per-client byte counter. This is the only correct source: HAProxy cannot report transferred bytes without stick tables (bytes_out is per-stream and zeroed by stream_new, and keep-alive allocates a new stream per request, so it never accumulates). res.body_size only gives advertised Content-Length, useless for streamed responses — deliberately not used as a fallback rather than introducing a second, inconsistent byte source.
  • Breadth/shape — resource and section fan-out, observed via the existing SPOE agent.

Also removes the HAProxy peer (stick-table) listener. It was dead code: it blocked on any stick-table update with no volume logic and was never wired to anything.

Enforcement kill switch

POST /api/enforcement/stop halts every enforcing command analyzer-wide while detection keeps running. Deliberately placed at the command-issuing boundary (Broadcast + sendCommand) rather than inside one detector, so it covers all of them. It is checked before the dedup reservation — otherwise a suppressed block would mark the IP "recently blocked" and stay suppressed for the dedup TTL after enforcement resumed. One-way by design: resuming needs a config change and a restart.

Safety

Everything ships off by default, and enabling detection does not enable blocking (-sustained-enabled vs -sustained-enforce). Design choices that bound false-positive cost:

  • Reputation multiplies request/byte floors only, never resource/section floors — breadth isn't an entitlement.
  • Release from hold is judged on requests/resources/sections, never bytes, since blocking destroys the byte evidence.
  • The hard hold ceiling is mandatory; good-reputation clients leave at the floor.
  • Allowlisted IPs are skipped entirely. The reputation hook is wired to verified bots, not the reputation engine — reputation entries are only created by penalties, so score 0 means "never seen", not "well behaved".
  • Raw paths and hostnames are never retained, only FNV-64a hashes with a separator so ("ab","c") != ("a","bc").

⚠️ Breaking change

-haproxy-port is removed. Go's flag parser rejects unknown flags, so a collector still passing it exits 2, and Restart=on-failure turns that into a crash loop. A local systemd drop-in overriding ExecStart= shadows the shipped unit and survives the upgrade — this is a real failure that happened during testing. collector-postinstall.sh now inspects the resolved config and names the offending files; it warns rather than edits, since the package has no business rewriting operator-authored drop-ins.

Check before upgrading:

systemctl cat packetyeeter-collector | grep -n haproxy-port   # expect no output

Housekeeping

Adds root-anchored ignore rules so stray go build -o analyzer output at the repo root can't be committed, and switches the docs to placeholder hostnames.

Validation

  • gofmt, go vet, git diff --check, make portable-test (13/13 packages) all clean.
  • eBPF verified through a real kernel verifier, not just compiled.
  • Validated on Linux 6.8 against live traffic: eBPF loaded and attached, egress_bytes maps populated per client, signals reached the analyzer attributed to the correct IP, the byte floor gated as designed (796 signals at a 4 KiB test floor vs 1 at the production 1 MiB floor), /api/sustained tracked hundreds of clients with blockers/margins populated, and the kill switch flipped both the endpoint and packetyeeter_enforcement_stopped.
  • Confirmed an older analyzer tolerates the new SIGNAL_EGRESS_VOLUME without erroring, so a collector-first rollout is safe.

Live testing also caught a bug not visible in unit tests: the collector serves metrics from an allowlist registry, so the egress counters were incremented but never scrapeable. They're now registered on the collector's registry, and the docs say which process to scrape (the analyzer registers the same names and reports a constant 0).

A code review during development caught one high-severity bug, fixed here with a regression test verified to fail against the old code: the IPv6 map iterator reuses one [16]byte key per poll, and signals are marshaled asynchronously off the queue, so storing the caller's net.IP directly meant a queued signal could report the wrong client.

awlx and others added 4 commits August 9, 2026 09:43
Remove the HAProxy stick-table peer listener. It was dead code: NewServer was
referenced only from its own e2e test, Config.HAProxyPort was assigned and
never read, and examples/haproxy.cfg has no peers section. README advertised a
feature that never ran.

Replace it with detection for a class of abuse rate limits cannot see. A
scraper that stays under every per-second limit but sustains that rate for an
hour across thousands of resources is invisible to a rate threshold and obvious
over a five-minute window.

Two independent selection paths over the same sliding window:

  volume - many requests, many resources, many sections, many bytes.
  shape  - breadth with no byte floor at all, gated by a resources-per-section
           ceiling. Enumeration is defined by breadth, so requiring bytes would
           mean only catching it once it also became a volume problem.

Breadth comes from the existing SPOE feed; host and path are already in
HTTPContext, so no HAProxy config change is needed. Bytes come from new eBPF TC
egress per-client counters, because HAProxy cannot report transferred bytes over
SPOE at all: its byte counters are stream-scoped, on-http-response fires before
the body moves, and a fresh stream is allocated per keep-alive request. The
egress counters see real wire bytes including streamed and chunked responses.

Hostnames and paths are hashed on the way in and never retained - the tracker
only counts and compares distinct values.

Reputation raises request and byte floors rather than exempting, and does not
raise breadth floors, since breadth is not an entitlement. Selected clients are
held, because blocking destroys the byte evidence that selected them; release is
judged on requests and breadth only, and a hard ceiling bounds the hold because
once enforcement removes the evidence nothing separates a false positive from a
suppressed true positive.

Detect-only by default, with -sustained-enforce a separate flag, and
/api/sustained reporting per client which thresholds it is under and how close
it is to each.

Also add an analyzer-wide runtime enforcement kill switch at the point commands
are issued (POST /api/enforcement/stop). It covers every detector, not one:
"we are blocking traffic we should not be" is not known up front to be confined
to a single detector, so an operator should not have to identify the culprit
before they can stop it. Relieving commands are not suppressed, since the switch
is pulled precisely when a block is wrong. One-way; resuming needs a restart.

Validation: gofmt, go vet, make portable-test, and targeted tests all pass on
macOS. The eBPF object was compiled with clang and loaded through a real kernel
verifier in Docker (all three programs, all 26 maps), but on a 7.0 kernel rather
than 5.4, and no live-traffic test was possible from this host. make collector,
make test, and the TC/XDP attach paths were not run.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Production rollout surfaced two gaps.

The collector serves metrics from an allowlist registry, so the egress
accounting counters were incremented but never scrapeable. The analyzer
registers the same names and reports a constant 0 for them, so there was
no way to tell "accounting is off" from "nothing crossed the floor" --
exactly the distinction the two counters exist to make. Register them on
the collector's registry and say in the docs which process to scrape.

Removing -haproxy-port is a breaking upgrade: Go's flag parser rejects
unknown flags, so a collector still passing it exits with status 2, and
Restart=on-failure turns that into a crash loop. A systemd drop-in that
overrides ExecStart= shadows the main unit, so editing the unit alone is
not enough -- document `systemctl cat` as the check that catches it.

Validated on production hosts (kernel 6.8, live traffic): eBPF egress
maps populate per client, signals reach the analyzer and are attributed
to the right IP, the byte floor gates as designed, and the analyzer-wide
kill switch flips /api/enforcement and packetyeeter_enforcement_stopped.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
The deb replaces the unit file, so removing -haproxy-port from the
shipped ExecStart= is enough for a stock install. It is not enough when a
local drop-in overrides ExecStart=: the drop-in shadows the unit and the
upgrade leaves it untouched, so the flag is still passed.

Go's flag parser rejects unknown flags, so the collector exits 2 and
Restart=on-failure turns that into a crash loop. Because postinstall
deliberately does not restart the service, the breakage surfaces at the
next restart, detached from the upgrade that caused it.

Check the resolved configuration in postinstall and name the offending
files. The package cannot safely edit operator-authored drop-ins, so this
warns rather than modifies, and does not fail the install.

Verified on a host carrying such a drop-in: silent when the resolved
config is clean, names the drop-in when it is not.
Stray `go build -o analyzer ./cmd/analyzer` output at the repo root is not
covered by the existing ignore rules, which only match the packetyeeter-*
names. Add the bare names anchored to the repo root -- unanchored
`analyzer` and `collector` would also match the pkg/ and cmd/ source
directories. Verified that no tracked file becomes ignored.

Switch the docs to example.com hostnames and a documented $COLLECTOR_HOST
variable so the commands are copy-pasteable against any deployment.
@awlx
awlx force-pushed the awlx-sustained-download-detection branch from 4cc4a95 to fe546a9 Compare August 9, 2026 07:47
Records the new detector, the analyzer-wide enforcement kill switch, the
removal of the dead HAProxy peer listener, and the breaking -haproxy-port
removal with the drop-in check operators need before upgrading.
@awlx
awlx merged commit a16236a into main Aug 9, 2026
4 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant