Skip to content

fix(network): report and retry a psn receiver or osc listener that is down - #484

Merged
michelhonold merged 5 commits into
mainfrom
fix/input-plane-failures
Oct 24, 2026
Merged

michelhonold merged 5 commits into
mainfrom
fix/input-plane-failures

Conversation

@michelhonold

@michelhonold michelhonold commented Oct 21, 2026 •

Copy link
Copy Markdown
Contributor

Summary

Fixes #421.

Two input planes could stay off after a failure without being reported:

  • PSN receiver. The network plane observer judged the PSN plane by its sender alone. If the receiver failed to bind at startup, the plane still read ok. No row was raised and the receiver was never started again.
  • OSC input. The OSC input plane was only followed while its listener was running. If a restart failed (port taken), the listener was left unset and the plane dropped out of the observer. OSC input then stayed off with no row until the next OSC save.

A plane that is not running is now reported as failing and retried with the observer's existing backoff (2 s, doubling to 60 s), whichever part of it failed. The rows use the existing plane-failure format, <plane>: <interface> – <reason>. There is no new status level or row type.

Plan

  • Observer: optional Plane.complete. A plane bound at the right address with one part not up is rebound like an unbound plane.
  • PSN plane: complete is false while the enabled receiver does not hold the sender's address. When there is no sender (PSN output failed to start), current is the receiver's address.
  • OSC input plane: whether it is followed now reads config only (enabled, a group, a pin). Apply and suspend rebuild a listener that is not running.
  • InputManager.resume_osc: rebuilds OSC input from the live config and raises while the listener cannot bind. restart_osc still logs and carries on for the config dispatcher.
  • Docs: network and input guides, Interface Assignment help drawer.

Notes

  • While the receiver keeps failing, PSN output keeps sending untouched. The receiver is rebound first and raises before the sender is reached. When the receiver does bind, the orchestrator rebinds both, so PSN output restarts once at that moment.
  • When PSN output failed to start, the receiver used to be torn down and rebound on every poll, because the plane always read unbound. Taking the receiver's address as the plane's address stops that.
  • The CLAUDE.md invariants hold. The OSC listener still binds the wildcard and a membership is still released by closing the socket. Planes still fail closed. All of this runs on the housekeeping tick (GTK thread), the same place config live-apply calls restart_osc from.
  • Still not reported: an OSC listener that fails to bind on a station where the OSC input row has no interface to follow (unpinned, or no multicast group). There is no plane in that case, and reporting it would need a new status row.

Test plan

  • Observer: a bound plane with a part not up is rebound, its failure reads as failing on the HUD and in the snapshot, and the retry waits out the backoff (tests/test_network_observer.py)
  • PSN plane: a receiver that never bound is a failing row and output keeps sending. It is not retried inside the backoff and recovers afterwards. With PSN input switched off the plane is not touched. A bound receiver with no output is left alone (tests/test_services_startup.py)
  • OSC input plane: still a plane while its listener is down. Apply and suspend rebuild the listener (suspend without a membership). A taken port raises, shows as a row, is retried with the backoff and recovers with its membership. A running listener is not rebuilt (tests/test_services_startup.py)
  • resume_osc rebuilds from the live config, raises on a bind failure, and drops the path and operator-message adapters with their cards (tests/test_input_manager_remaining.py)
  • Mutation-checked: 13 mutants, 12 killed. The survivor was an equivalent check, now removed
  • 100 % line and branch coverage of changed lines. ruff, ruff format and mypy clean
  • make ci-remote on the testing Pi
  • Station: a PSN receiver whose bind fails at startup shows the PSN row and recovers on its own once the bind succeeds
  • Station: with the OSC input row pinned and a multicast group set, OSC input on a port taken by another process shows the OSC input row, and OSC input comes back with its membership once that process lets go
    • Checked with OSC input following the station interface (Simple mode) and its multicast group set.

Out of scope: an OSC listener that fails while OSC input follows no interface (unpinned on auto-detect, or no multicast group) has no plane to report it; that is #481.

🤖 Generated with Claude Code

michelhonold and others added 5 commits October 21, 2026 00:25
… down

The network plane observer judged the PSN plane by its sender alone, so a
PSN receiver that failed to bind at startup left the plane reading ok: no
row, no retry, no viewer markers until the next interface change. The OSC
input plane was followed only while its listener ran, so a listener that
failed to restart (port taken) dropped out of the observer altogether and
stayed off until the next OSC save.

A plane now gets an optional `complete` check: a plane bound at the right
address with a part not up is rebound like an unbound one, so the failure
is a failing row retried with the observer's backoff. The PSN plane reads
false there while its enabled receiver does not hold the sender's address;
with no sender (PSN output failed to start) the receiver is the plane, so a
bound receiver is no longer rebound once a second.

Whether the OSC input plane is followed now reads config only. Its apply
and suspend rebuild a listener that is not running from the config through
the new `InputManager.resume_osc`, which raises while the port is taken,
where `restart_osc` keeps logging and carrying on for the config dispatcher.

Fixes #421

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…cards

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A switched-off receiver never binds, so its bound address already reads None.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…cribe

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@michelhonold michelhonold added bug Something isn't working Network Multi-interface networking: per-plane interface pinning, binding, and runtime IP handling labels Oct 21, 2026
@michelhonold

Copy link
Copy Markdown
Contributor Author

Review rounds

Each round was a fresh, full review of git diff origin/main HEAD.

Round 1

  • Medium (docs). In network.md, the new OSC sentence sat in front of "That rebinds the listener", so "That" pointed at the wrong subject. Moved the sentence after the membership-release explanation. Fixed in fbdc3ff.
  • Medium (help copy). The help drawer said every function that cannot start shows a row and is retried. That is only verified for PSN and OSC input; the video plane's apply may not raise. Narrowed the paragraph to PSN; the OSC sentence stays in its own section. Fixed in fbdc3ff.
  • Low (mutation survivor). The receiver.enabled check in the PSN plane's no-output fallback could never change the result, because a switched-off receiver never binds. Removed. Fixed in 37c2525.
  • Low (test gap). The finally that rebuilds the operator-message and path adapters after a failed resume was not pinned. Added a check that the cards are dropped. Fixed in beee002.

Round 2

  • Low (docs). The input guide named only restart_osc as resetting the OSC stream log. resume_osc does too. Fixed in 76eede4.
  • Checked with no finding: the PSN suspend re-check (a receiver restarted elsewhere while the plane is suspended is now stopped again, where before it was missed), outage recovery still rebuilding both PSN sides, the OSC repin-failure path through the real OscService/OscMarkerAdapter state, and branch coverage of every changed line.

Round 3

Nothing beyond cosmetic or rare.

Cosmetic, left as is

  • While the receiver keeps failing, each retry also runs the orchestrator's rollback to the same address. That is a second bind attempt and a rollback traceback in the journal per retry, bounded by the backoff.
  • The PSN row's reason is the receiver's own exception text (PSN receiver failed to bind to source_ip='…'). Existing wording, but technical.

Rare, left as is

  • The OSC listener dies while its interface is down and the plane is already suspended. It is only restarted when the interface returns. The is down row is showing the whole time.

Results

  • Targeted suites: 841 passed. ruff, ruff format and mypy clean.
  • Mutation: 13 mutants, 12 killed. The survivor was an equivalent check, now removed.

@michelhonold
michelhonold marked this pull request as ready for review October 24, 2026 19:25
@michelhonold
michelhonold merged commit c7c359b into main Oct 24, 2026
4 checks passed
@michelhonold
michelhonold deleted the fix/input-plane-failures branch October 24, 2026 19:25
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working Network Multi-interface networking: per-plane interface pinning, binding, and runtime IP handling

Projects

None yet

Development

Successfully merging this pull request may close these issues.

fix(network): a PSN receiver or OSC listener that fails to start stays off unreported

1 participant