§7 hardening before any public post: the coordinator's bind has no private network to hide behind

[defenwycke]

Reconnaissance done 2026-09-06, before the v0.20.0 Delving/HN post. RUNBOOK.md §7 lists what must be closed before a public write endpoint is announced. One item does not fit the architecture as written.

What §7 assumes vs what exists

§7 says: "If the coordinator is a separate box, set COORD_BIND to its private-network IP (not 0.0.0.0) and firewall :8899 to the web box."

There is no private network. The coordinator's interfaces are:

lo        127.0.0.1/8
eth0      152.53.93.164/22     <- public
docker0   172.17.0.1/16

The web box (83.136.255.218) proxies to the coordinator over the public internet. So:

• COORD_BIND=127.0.0.1 would break the proxy — the board would go dark
• there is no private IP to bind to instead
• COORD_BIND=0.0.0.0 is currently the only workable value

What is actually protecting it

Measured, with a working control:

probe	result
152.53.93.164:8899	curl_exit=28 — timed out, i.e. filtered
152.53.93.164:9110	curl_exit=7 — refused
bitcoinghost.org/hazync/api/meta (control)	http=200 — the probe method works

So a provider-level filter drops :8899 from arbitrary hosts while allowing the web box. That is real protection, but it is one layer, and it is not ours — ufw is inactive and iptables INPUT is empty on the host.

Proposed

1. Host firewall allowing :8899 only from the web box's egress address — defence in depth that does not need a private network, and survives a provider-side rule changing.
2. Keep COORD_BIND=0.0.0.0 and correct §7's wording, which currently prescribes something impossible on this topology.
3. The rest of §7 (nginx limit_req/limit_conn, the 1s /api/state micro-cache, client_max_body_size 8m, TRUSTED_PROXIES) applies on the web box and is unaffected.

⚠ Not changed yet, deliberately: altering live network access mid-release risks the board. Do it with the cutover restart, when the coordinator is already down.

Blocks: the Delving/HN post. An anonymous GET flood on /api/state is, in §7's own words, "the cheapest board-takedown".

[defenwycke]

Corrected by measurement. §7 is nearly all done already — I filed this as though it were largely outstanding, and it is not.

Verified in place, on the live system

§7 item	state
limit_req / limit_conn	✅ live — hazync_api at 10r/s, hazync_conn 20
/api/state micro-cache	✅ live — proxy_cache hazync_cache, proxy_cache_lock on, 1s validity
cache dir	✅ /var/cache/nginx/hazync, owned www-data:www-data
client_max_body_size 8m	✅ set on the submit path (256k elsewhere)
TRUSTED_PROXIES	✅ 83.136.255.218 — the web box specifically, not a wildcard
VERIFY_MODE	✅ real, not mock

And the rate limit was tested, not merely read. 60 parallel requests to /api/state:

33 x 503
27 x 200

A first attempt with sequential curl returned 45/45 × 200 and proved nothing — the per-request overhead held it near 6 r/s, under the limit. That was a check that could not fail; the parallel run is the one that means something.

⇒ The "anonymous GET flood on /api/state is the cheapest board-takedown" vector is closed and demonstrated.

What actually remains

Only the item this issue was really about: COORD_BIND=0.0.0.0 with no host firewall. As established above, §7 prescribes binding a private IP and there is no private network — the coordinator has lo, a public eth0, and docker0. So the remaining work is:

1. a host firewall allowing :8899 only from 83.136.255.218 — defence in depth, since today the only layer is a provider filter that is not ours (ufw inactive, iptables INPUT empty)
2. correcting §7s wording, which prescribes something impossible on this topology

Neither blocks a public post the way an open flood vector would. Downgrading this from "blocks the Delving/HN post" to "should be done before it, and is one command plus a doc fix".

[defenwycke]

Two corrections to the reconnaissance above, and one item actioned.

1. ufw is not inactive — the opposite

This issue says "ufw is inactive and iptables INPUT is empty on the host". Measured 2026-09-08:

ufw: active, default deny (incoming)
-P INPUT DROP
-A INPUT -s 83.136.255.218/32 --dport 8899 -j ACCEPT   <- web box
-A INPUT --dport 8899 -j DROP

So proposal (1), a host firewall restricting :8899 to the web box, already exists. The provider-level filter is no longer the only layer.

⚠ But there are two sources of truth for one port: hand-added rules sit directly in INPUT, ahead of ufw's chains, and ufw status does not show them. Read iptables -S INPUT as well, or you will trust a rule something else overrides.

2. A stale rate-limit bypass, which is worse than the hole this issue describes

RATE_EXEMPT listed 94.237.17.228 — added 2026-07-27 as "our own prover", an UpCloud box in the same range as the retired hazync-b200 (94.237.18.83). It was gone: 0 packets in 90 s against the web box's 1,690, no connection on any port, not on the account.

That is not just reachability. An exemption is unlimited access to /api/state for whoever the provider issues that address to next — precisely the flood this issue calls "the cheapest board-takedown".

⚠ I first called it stale on packet counts alone, which was wrong reasoning — it is referenced in two places and described as ours, so it was deliberate, not forgotten. A dormant counter shows silence, not decommissioning. What settled it was the account inventory.

Actioned (#232): iptables ACCEPT removed, exemption removed, both backed up (/root/iptables.bak.20260908, /root/ratelimit.conf.bak.20260908). Verified after: public /api/meta still served through the web box, frontier advanced 3,644 → 3,646 across the restart, workers still crediting, public GETs 200.

3. Proposal (2) done in #230

§7's "set COORD_BIND to its private-network IP" was unfollowable — there is no private network, and 127.0.0.1 would take the board dark. The RUNBOOK now says 0.0.0.0 is correct here and that the control is the firewall, not the bind.

Still open

The web-box half of §7 — nginx limit_req/limit_conn, the 1 s /api/state micro-cache, client_max_body_size 8m, TRUSTED_PROXIES. Untouched, and still what gates the public post.

[defenwycke]

§7 walked end to end. All three bullets now verified rather than assumed.

§	item	state
1	nginx rate/conn limits + micro-cache	✅ deployed and firing — 40 concurrent GETs on /api/state gave 20×200 + 20×refused, matching burst=20 exactly; x-cache-status: HIT; nginx -t clean; no backups in sites-enabled/
2	secure bind	✅ wording corrected (#230), firewall confirmed restricting :8899 to the web box, stale ACCEPT + RATE_EXEMPT entry removed (#232)
3	trusted proxy	✅ TRUSTED_PROXIES=83.136.255.218 — set to the web box, not the default

Bullet 3 was the one with real teeth ("else the rate limit is bypassable") and the implementation holds up under the obvious attack:

• nginx sets proxy_set_header X-Forwarded-For $remote_addr on both hazync locations — overwrite, not append, so a client cannot inject a header to be attributed to someone else
• _client_ip() honours X-Forwarded-For only when the direct peer is in TRUSTED_PROXIES, and falls back to the peer otherwise

So public traffic is limited per real client, not lumped under the proxy.

⚠ One residual worth a look, not changed here: 83.136.255.218 is in both TRUSTED_PROXIES and RATE_EXEMPT. Today that exemption is unreachable, because nginx always sets XFF for these locations, so _client_ip() never returns the proxy's own address. But it is a fail-open — if any future path proxied to :8899 without setting XFF, every request through it would be attributed to the proxy and exempted wholesale. Removing the web box from RATE_EXEMPT would make that fail closed instead. I have left it, because it needs checking against every path that reaches :8899, not just the two I looked at.

Also improved while verifying: the limiter answered 503, indistinguishable from a genuine outage. Now 429 (#232), re-measured 20×200 + 20×429, scoped to the hazync locations so ladder_api/pool_api are untouched.

⇒ Once #232 merges, I believe this issue is closeable and the Delving/HN post is no longer blocked on §7 — but that is a call for the operator, not for me.