Replication: no backoff or status signal on persistent auth failure — reconnect loop generated 73M log events/week, undetected

[kriszyp]

Problem

A Datadog audit (7 days, 2026-06-30 → 2026-07-06) found that ~36% of all indexed log volume is a single replication auth-failure reconnect loop:

message pattern	7d count
Disconnected from wss://<host>:9933 (db: "<db>"), due to "Unauthorized" (code: 1008)	34.2M
Connected to wss://<host>:9933, db: <db>	27.5M
Incoming client connection from ::ffff:<ip> did not have valid certificate, you may need turn on enableRootCAs...	16.8M
No node found for IP address ::ffff:<ip>, available nodes are <full node list>, connection will require credentials.	16.8M
No authorization provided	11.4M
MQTT auth failed: no credentials / not authorized { + multi-line object dump	12.1M

That's ~73M log events in a week from peers that connect, get rejected as Unauthorized, and immediately reconnect — continuously, for weeks. No monitor fired and nothing surfaced in get_status/cluster_status — the flap was only discovered by auditing Datadog billing data.

Three defects compound here:

1. No auth-aware backoff. A persistent auth/authorization rejection (WS close 1008, missing credentials, unknown-node) is retried on the same tight schedule as a transient network error. Auth failures don't self-heal in milliseconds; retrying instantly just burns CPU, connections, and log volume.
2. Every cycle logs 3–6 lines at warn/error, including the full node list in the No node found for IP message (unbounded message size on large clusters) — with no dedup (harper-pro#262).
3. No durable status signal. A peer in a persistent auth-failure loop should be a single error-level status entry (per peer, with occurrence count and last-error detail), visible in get_status, not millions of identical log lines.

Proposed fix

• Classify connection failures: auth-class rejections (1008 Unauthorized, No authorization provided, unknown node/cert rejection) get exponential backoff with a high cap (e.g. 30s → 10min), distinct from transient network errors.
• One status entry per peer (e.g. replication.connection.<node>), set on failure with the close code/reason, cleared on successful authenticated connect — so the flap is visible and alertable via the status system (PR Add integrated logger-status system with .status() API, health checks and hierarchical view #372 direction).
• Log the first occurrence and every Nth (or once per backoff step) with a repeat count; drop the full node list from the hot-path message (log it once at debug).
• Surface the last connection error per socket in cluster_status (harper-pro#214 / ci(claude): tighten override so concise-PR rule beats universal calibration clause #437 Tier 1).

Related

harper-pro#437 (Replication W8 observability), harper-pro#262 (log dedup), harper-pro#214 (last connection error), PR #372 (logger.status bridge). Live flapping clusters tracked separately in harper-pro.

🤖 Filed by Claude on behalf of Kris.