hardening: connectivity robustness + internal-SRAM relief - #84
Open
iliabaranov wants to merge 7 commits into
Open
hardening: connectivity robustness + internal-SRAM relief#84iliabaranov wants to merge 7 commits into
iliabaranov wants to merge 7 commits into
Conversation
Enabling WiFi at runtime while the USB-NCM tether stack is resident (and/or mbedTLS holds its dynamic buffers during a Tailscale bring-up burst) exhausts internal DMA-capable RAM, and a WiFi-driver allocation ABORTS internally rather than returning ESP_ERR_NO_MEM -> PANIC + reboot (repro: two PANIC resets when WiFi was toggled on at ~73 KB free internal on an eth+USB-resident PSTOP54). Add a pre-flight internal-heap floor (WIFI_MIN_INTERNAL_HEAP, 90 KB) checked before esp_wifi_init; on a shortfall, synthesize ESP_ERR_NO_MEM and fall into the existing cleanup + 4 s deferred-retry path. This converts the OOM crash into a safe, retried deferral (fail-safe: no WiFi -> eventual STOP if it's the only uplink, never a crash-loop). Validated on PSTOP54: enabling WiFi (the exact action that PANIC-crashed it) now leaves rst_hist unchanged (no new PANIC), uptime monotonic (no reboot), and the safety bond held (state=2) throughout; restore clean. Threshold is conservative + tunable: set above the ~73 KB panic point, reachable once mbedTLS frees ~50 KB when TLS is idle. Needs HIL tuning for the WiFi-actually-associates case; the deeper fix (fitting WiFi alongside a resident USB-NCM stack) is the Internal-RAM Phase-3 work. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UJz4ZB7WcPErDqKLWDq3ht
…nternal) The 16 WiFi static TX buffers pre-reserve ~25 KB of DMA-locked internal RAM at esp_wifi_init; 6 is ample for the low-rate (10 Hz, tiny-frame) pstop link and returns ~16 KB, so WiFi init leaves ~39 KB free instead of the ~23 KB that OOM-panicked before (the real fix behind the WIFI_MIN_INTERNAL_HEAP guard, which is re-tuned 90 -> 72 KB to match the smaller footprint). TX-side only, by design: TX buffers do not gate the WPA2 EAPOL handshake, so this avoids the RX-ring starvation that broke association in an earlier trim. Dynamic TX was not an option — ESP_WIFI_DYNAMIC_TX_BUFFER depends on !SPIRAM_TRY_ALLOCATE_WIFI_LWIP and we keep WiFi lwIP in PSRAM (a bigger win). Builds with STATIC_TX_BUFFER_NUM=6; the ~16 KB is realized at esp_wifi_init (static WiFi buffers aren't allocated until WiFi starts, so no idle delta on an eth-only unit). docs/INTERNAL_RAM_REDUCTION.md Phase 3 marked done (safe half). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UJz4ZB7WcPErDqKLWDq3ht
… with remote) Mirror firmware/sdkconfig.defaults on the machine app so machn gets the same ~16 KB internal-DMA reduction at esp_wifi_init (it shares the dcs_support dcs_wifi OOM guard). TX-side only; RX ring left at stock. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UJz4ZB7WcPErDqKLWDq3ht
The on_usb_rx per-frame copy handed to lwIP was malloc'd from internal SRAM on the device-side USB-NCM path — the scarcest heap on the S3, and the pool that dipped to an ~8 KB low-watermark on the tethered device. The buffer is filled by a plain CPU memcpy on the TinyUSB task (never DMA, never ISR) and consumed on the TCPIP thread, so the flash-cache- disable hazard does not apply; move it to PSRAM via heap_caps_malloc. This removes per-frame internal-heap alloc/free churn (fragmentation of the last few internal KB) from the tether RX hot path. netif_l2_free()'s free() handles a PSRAM pointer unchanged. The dominant USB-NCM consumer (the ~19.3 KB NTB DMA block) is pinned to internal SRAM by the S3 USB-OTG DMA engine and cannot move; reclaiming it via NTB count/size trim is tracked separately, gated on the USB-NCM stability regression. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UJz4ZB7WcPErDqKLWDq3ht
…hine timeout) The per-session reply-loss watchdog was set to exceed the machine's heartbeat timeout so the remote only re-bonds AFTER machn would itself have dropped the bond — never on a sub-timeout reply blip. That invariant broke when max_missed was raised 3 -> 5 (2026-08-04): the machine timeout moved 1000 -> 2000ms but REBOND_AFTER_MS stayed 1500ms, dropping it BELOW the timeout. Consequence (root-caused from a soak disconnect): when a second remote joins the tailnet, the peer-join X25519/DISCO crypto burst time-shares the wg_mgr task that also decrypts the safety-heartbeat replies, lagging them ~1.6s. At 1500ms that lag forced a nuisance rebond even though machn still held the bond (gap 49->57, sf_txdrv=0, mismatch=0 — replies merely late, not lost). 2500ms (= 2000 + 500 jitter) restores "rebond only after a real machine-side drop." Safety unaffected: machn's 2.0s timeout remains the STOP authority; this constant governs only connectivity re-sync. Immediate mitigation for the cross-remote disruption; the deeper fix (priority-isolate the safety decrypt from control-plane crypto) is designed separately for review. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UJz4ZB7WcPErDqKLWDq3ht
…rv under load) A 3-device soak at ~70% core-1 load showed a remote on USB-NCM hitting pstop_sf_txdrv=9 (unrecovered TX drops) and one self-rebond. Root cause: chip TX over USB-NCM refuses (ERR_IF, can_xmit=false) when all IN NTBs are in-flight awaiting the host's bulk-IN poll; the only transient recovery is the sess_sendto retry loop, and at high load the prio-5 CPU1 TinyUSB task is slow to drain NTBs, so bursts outran the old ~3 ms (3×1 ms) window. Raise PSTOP_TX_RETRY_MAX 3 -> 6 with a 1 ms→2 ms backoff (~9 ms worst case). ERR_IF-gated only — a dead link (route/ENOMEM) still fails on attempt 0, so dead-link STOP latency is unchanged; ~9 ms is still << the send period and << the machine's 2.0 s timeout. Zero internal-RAM cost (unlike raising the IN NTB count, the secondary lever held in reserve since it re-pressures the Phase-3 internal-RAM budget). Also corrects the stale "~1.2 s timeout" comment to the current 2.0 s (400 ms × max_missed 5). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UJz4ZB7WcPErDqKLWDq3ht
…fety-decrypt isolation) Root cause of a both-remotes-simultaneous rebond under tailnet churn: wireguardif_periodic() loops all WIREGUARD_MAX_PEERS (128) and runs a full X25519 handshake initiation (~30 ms) for EVERY peer that needs one, in a single uninterruptible call. When a netmap resync makes many peers re-handshake at once on a full peer table, that one call runs multiple seconds — and ml_wg_mgr drains the safety-heartbeat reply queue only BETWEEN calls, so both bonded remotes' replies starve for >2.5 s and both rebond (machn itself never rebooted). Fix (Option B, user-approved): budget the expensive handshake INITIATIONS to 4 per call, round-robin across calls (static resume index) so every peer is still serviced, deferring the rest to the next call. Cheap per-peer work (reset / keepalive / link_up) still runs for all peers each call. An established bond has no initiation pending, so the pinned safety peer is never delayed by the budget. Caps each call to ~120 ms of X25519 << the 2.0 s heartbeat timeout / 2.5 s reply-loss threshold, so ml_wg_mgr's between-call drain keeps the safety decrypt fresh. No API change; no dropped handshakes (deferred, retried next call). Shared wireguard_lwip -> applies to both remotes and machn (the both-remote stall was machine-side). Complements REBOND_AFTER_MS 2500 (which alone couldn't cover a >2.5 s stall). Broader mitigation: deploy the Tailscale ACLs to cut the churn that drives the handshake storm (fleet-side, tracked separately). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UJz4ZB7WcPErDqKLWDq3ht
iliabaranov
changed the title
This was referenced Aug 8, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Consolidates three closely-related low-risk hardening themes into one PR (123 code lines total). No file overlap between the groups; all cherry-picked clean off
mainand build clean together (ESP-IDF 5.5,idf.py buildexit 0).Group 1 — connectivity robustness (was #83; soak-validated)
main.c:REBOND_AFTER_MS1500 → 2500 ms. The self-rebond interval must exceed the machine's liveness timeout (heartbeat_ms × max_missed= 400 ms × 5 = 2.0 s); at 1500 ms a healthy remote could re-bond inside the window and momentarily present as a fresh remote. Restores the invariant aftermax_missedwent 3 → 5.main.c: pstop TX retry budget 3 → 6 with backoff. USB-NCMsess_sendtoreturnsERR_IF/sf_txdrvunder load; the wider budget with a1→2tick backoff rides out transient TX-driver stalls without dropping a heartbeat.wireguardif.c: pacewireguardif_periodichandshake initiations (Option B). The unbounded per-tick loop over all 128 peer slots could burst X25519 handshakes and starve the safety-decrypt path; round-robin with a per-tick budget of 4 bounds it. Validated: 5h+ into a 12h zero-disconnect soak with GREEN held and 0 connectivity rebonds (this is the change that soak is exercising).Group 2 — WiFi internal-SRAM (this PR's original scope)
dcs_wifi.c: guardesp_wifi_initagainst OOM panic. With USB-NCM resident, internal DMA RAM could be exhausted and a WiFi-driver alloc would abort (panic+reboot). Pre-flight internal-heap floor → defer via the existing 4 s retry path instead of crashing.sdkconfig.defaults(+machn/): static TX buffers 16 → 6. Frees ~16 KB internal DMA; ample for the low-rate pstop link. RX ring left at stock to protect the WPA2 EAPOL handshake.Group 3 — USB-NCM RX copy → PSRAM (was #85)
ml_dev_tether.c:on_usb_rxcopymalloc→heap_caps_malloc(MALLOC_CAP_SPIRAM). CPU memcpy on the TinyUSB task (not DMA, not ISR) consumed on the TCPIP thread → cache-disable hazard N/A. Removes per-frame internal-heap churn on the device-side USB-NCM path (the device that dipped to an ~8 KB internal watermark). The dominant ~19.3 KB NTB DMA block is pinned to internal SRAM by the S3 USB-OTG DMA engine and cannot move — reclaiming it by NTB count/size trim is tracked separately, gated on the USB-NCM stability regression.Verification
Builds clean together; each change independently verified. Hardware watermark/soak validation for Groups 2–3 to follow once the in-flight 12 h soak completes (so it isn't perturbed). Supersedes #83 and #85.
🤖 Generated with Claude Code