Skip to content

feat(debug): live device screen mirror with remote control (protobufs #1054) - #6987

Merged
jamesarich merged 26 commits into
mainfrom
screen-mirror-poc
Oct 29, 2026
Merged

jamesarich merged 26 commits into
mainfrom
screen-mirror-poc

Conversation

@jamesarich

@jamesarich jamesarich commented Aug 31, 2026 •

Copy link
Copy Markdown
Collaborator

The Debug panel can only show what the app sees. This adds a Mirror tab that shows the connected device's own screen live and drives it: D-pad, keyboard, swipe, and touch on panels that have it. It consumes DisplayFrame, DisplayPalette and DisplayInfo from protobufs#1054, now merged and pinned as 2.8.1.22-gc858ad1-SNAPSHOT.

Verified on hardware against a RAK4631 (mono OLED) and a LILYGO T-Deck (color, touch). The color path is MUI dirty rects and needs device-ui#394, which has not shipped, so a build from these branches gets the mono path only.

🌟 New Features

  • Live screen mirror: FromRadio.display_frame chunks reassembled into frames, rendered 1:1 into an ImageBitmap and upscaled nearest-neighbor. Color devices stream a signature-keyed DisplayPalette; per-region RGB565 colors apply with last-region-wins, and a frame that arrives before its palette turns color when the palette lands.
  • Remote control three ways: a 5-way D-pad ring (hold-to-repeat, long-press OK sends SELECT_LONG, haptics, screen-reader semantics); keyboard capture through core/ui's shared RemoteKeyboardSink (soft keyboard on Android, one kb_char per code point, arrows navigate, Enter is OK, Esc is Back, Ctrl/Alt chords stay with the host); swipe on the image, and tap or long-press forwards panel coordinates on has_touch devices.
  • Adaptive layout: controls beside the mirror above 760 dp, below it otherwise.
  • DeviceMetadata.display gates the feature: only mirror-capable firmware fills it, so without it the tab shows an unsupported state and refuses to arm or request frames, while the D-pad stays live because send_input_event predates mirroring. E-ink devices get a one-shot Refresh hint, touch enables tap-to-touch.

🛠️ Refactoring & Architecture

  • core/repository: DisplayMirrorManager with MirrorFrame and MirrorPalette; AdminController.setDisplayMirror, requestDisplayFrame and sendInputEvent send immediately so input never queues behind bulk admin traffic.
  • core/data: the reassembly state machine. Contiguous offsets only, geometry pinned at the first chunk, size and region caps checked in Long before any allocation, reset() so a previous device's last screen is never shown as live. MirrorRectCompositor holds the dirty-rect path.
  • core/service: the three admin senders over sendAdminImmediate.
  • feature/settings: the Mirror tab (DisplayMirror.kt, DisplayMirrorViewModel.kt, DpadRing.kt, MirrorFrameRenderer.kt). The stream stops on tab dispose, ViewModel clear and disconnect. Managed mode refuses the admin write and input injection in the ViewModel and disables the controls in the UI.
  • Strings localized in core/resources; two Material Symbols chevrons added.

🧹 Chores

  • org.meshtastic:protobufs to 2.8.1.22-gc858ad1-SNAPSHOT, the first master snapshot carrying [Feature Request] Send any CLI command  #1054. The mavenLocal pin this branch carried is gone.
  • Rebased onto main; the focus clear is a keyed SideEffect now that type-resolved detekt flags a non-suspending LaunchedEffect.

Testing Performed

  • DisplayMirrorManagerImplTest: frames, palettes, drops, restarts, overflow, the caps, and a region_total fixed per palette.
  • FromRadioPacketHandlerImplTest: display_frame and display_palette routing.
  • MirrorColorResolverTest: region precedence and RGB565 bit replication.
  • DisplayMirrorInputTest: one code point per kb_char, surrogate pairs, and which host keys map to device events.
  • DisplayMirrorViewModelTest: no arm or frame request without DeviceMetadata.display, both with it, keys either way, and the gate opening when metadata lands.
  • Baseline spotlessApply spotlessCheck detekt detektTypeResolved assembleDebug test allTests kmpSmokeCompile -PwarningsAsErrors=true on the rebased branch.

Known gaps, unfixed here: frame-render cost on the composition thread, and rect-reassembly panel geometry across continuation chunks.

Feature stack

Release order runs top to bottom.

Repo PR Role
meshtastic/design #142 Cross-platform feature spec
meshtastic/device-ui #394 DisplayMirror: frame capture and remote input
meshtastic/protobufs #1054 DisplayFrame, DisplayPalette, DisplayInfo and the two admin verbs (merged)
meshtastic/firmware #11681 Producer: streams the framebuffer, bridges input
meshtastic/Meshtastic-Android #6987 Mirror tab with remote control
meshtastic/meshtastic-mcp #78 capture_display tooling

device-ui#394 is the blocker for the color path; until it lands and a pin bumps, every committed firmware configuration compiles the MUI path out.

Summary by CodeRabbit

  • New Features
    • Added a Mirror tab in the Debug Panel to view a connected device’s screen in real time.
    • Use directional controls, keyboard input, or touch gestures to interact with the device screen. Available controls depend on device capabilities and connection status.
    • View monochrome and color display frames, including supported display palette colors.
    • Enable mirroring or request a single frame from the mirror controls.

@coderabbitai

coderabbitai Bot commented Aug 31, 2026 •

Copy link
Copy Markdown
Contributor

Review in Change Stack →

Note

Currently processing new changes in this PR. This may take a few minutes, please wait...

Run configuration
Commits

Reviewing files that changed from the base of the PR and between 6d690aa and dd56444.

Files selected for processing: 5
  • core/resources/src/commonMain/composeResources/values/strings.xml
  • core/testing/src/commonMain/kotlin/org/meshtastic/core/testing/FakeRadioController.kt
  • feature/settings/src/commonMain/kotlin/org/meshtastic/feature/settings/debugging/DisplayMirror.kt
  • feature/settings/src/commonMain/kotlin/org/meshtastic/feature/settings/debugging/DisplayMirrorViewModel.kt
  • feature/settings/src/commonTest/kotlin/org/meshtastic/feature/settings/debugging/DisplayMirrorViewModelTest.kt
Walkthrough

Adds device-screen mirroring to Debug. The change introduces frame and palette models, packet reassembly, local-node controls, and a UI for displaying mirrored frames and sending keyboard, touch, and remote-control input.

Changes: Device Screen Mirroring

Layer / File(s) Summary
Mirror contracts and device controls
core/repository/.../DisplayMirrorManager.kt, core/repository/.../AdminController.kt, core/service/.../AdminControllerImpl.kt, core/testing/.../FakeRadioController.kt
Adds frame and palette models and manager contracts. Adds immediate local-node operations to enable mirroring, request a frame, and send input events.
Packet routing and chunk reassembly
core/data/.../DisplayMirrorManagerImpl.kt, core/data/.../MirrorRectCompositor.kt, core/data/.../FromRadioPacketHandlerImpl.kt, core/data/src/commonTest/.../manager/*Test.kt
Routes display packets from active sessions. Assembles monochrome frames and palettes, composes RGB565 rectangles, and tests chunk validation, routing, and reset behavior.
Mirror state, rendering, and input
feature/settings/.../debugging/*, feature/settings/.../radio/RadioConfig.kt, feature/settings/src/commonTest/.../debugging/*, core/resources/..., core/ui/.../Navigation.kt, .skills/compose-ui/strings-index.txt
Adds mirror state and controls, frame rendering, input handling, and the Mirror Debug tab. Adds related strings, arrow icons, and tests for input mapping and color resolution.

Priority: Normal

Merge Risk: Low · previous review 6d690

Mirroring can briefly show inconsistent or stale state when a reset overlaps an incoming frame. The remaining risk is bounded, but synchronizing reset and packet handling is advisable.

Pre-merge checks: Passed 5 · Failed 3

Failed Failed checks: 3 warnings

Check name Status Explanation Resolution
Tests Prove The Path, Not The End State Warning The standing failure remains unresolved. DisplayMirrorManagerImplTest still has end-state-only assertions: single-chunk frame completes immediately checks only pixel count (lines 85-90); the two-c… Strengthen the affected tests. Assert exact frame bytes and metadata for the single-chunk frame. Use distinct palette regions and compare the complete ordered region list, including coordinates and colors. Assert RGB565 bytes at expected re…
Regression Coverage For Changed Behavior Warning Regression coverage remains incomplete for changed behavior. The diff adds only four test files: manager/radio routing tests, input helper tests, and color-helper tests. The following gaps remain: - `… Add focused AdminControllerImpl protobuf/destination tests; DisplayMirrorViewModel fake-flow lifecycle and managed-mode tests; valid multi-chunk RGB565 pixel and canvas-preservation tests; remaining malformed frame/palette boundary and …
Moved Code Diffed Against Its Original Warning The visibility widening remains. Commit 4247323 moved private MirrorFrameImage from DisplayMirror.kt to internal MirrorFrameRenderer.kt:60. Commit e31e2d1 moved private input helpers and con… Preserve the original private scope. Keep MirrorFrameImage and utility helpers in their caller file, or add private wrappers and test through the caller. Keep MirrorRectCompositor private or nested inside DisplayMirrorManagerImpl. If …
Passed Passed checks: 5 passed
Check name Status Explanation
Title check Passed The title clearly identifies the main change: a live device screen mirror with remote control in the debug panel. The protobufs reference is relevant supporting context.
Linked Issues check Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check Passed Check skipped because no linked issues were found for this pull request.
Sibling Call Sites And Presence Semantics Passed PASS. The diff does not change ambient temperature, humidity, RSSI, SNR, current, voltage, or particulate-field presence semantics. It adds nullable mirror state and nullable DisplayInfo; all `displ…
Description Check Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Full details: Sibling Call Sites And Presence Semantics

Explanation

PASS. The diff does not change ambient temperature, humidity, RSSI, SNR, current, voltage, or particulate-field presence semantics. It adds nullable mirror state and nullable DisplayInfo; all displayInfo consumers are in the new mirror UI and use null-safe checks. The only zero defaults are mirror protocol state, palette colors, and input coordinates, not physical measurement fields covered by this check.

Full details: Tests Prove The Path, Not The End State

Explanation

The standing failure remains unresolved. DisplayMirrorManagerImplTest still has end-state-only assertions: single-chunk frame completes immediately checks only pixel count (lines 85-90); the two-chunk palette test uses the same region twice and checks only regions.size (lines 154-188); RGB565 composition checks only format and canvas size, not pixel placement or untouched bytes (lines 284-291); and the surplus-palette test checks only collection size with identical regions (lines 342-359). The reset test seeds only completed mono and palette values, so it does not prove that incomplete mono, palette, or RGB565 state was cleared (lines 217-225). These tests could pass with incorrect bytes, region contents, or retained partial state.

Resolution

Strengthen the affected tests. Assert exact frame bytes and metadata for the single-chunk frame. Use distinct palette regions and compare the complete ordered region list, including coordinates and colors. Assert RGB565 bytes at expected rectangle coordinates and untouched canvas bytes. Assert the surplus-palette case against the exact published region list. For reset, seed incomplete mono, palette, and RGB565 rectangle sequences, call reset(), and verify that continuations do not complete until fresh offset-zero sequences begin.

Full details: Regression Coverage For Changed Behavior

Explanation

Regression coverage remains incomplete for changed behavior. The diff adds only four test files: manager/radio routing tests, input helper tests, and color-helper tests. The following gaps remain: - AdminControllerImpl.setDisplayMirror, requestDisplayFrame, and sendInputEvent (core/service/.../AdminControllerImpl.kt:147-171) have no implementation tests. A wrong local destination, protobuf field, input default, or sendAdminImmediate call can reach the radio. Add tests that capture the exact destination, protobuf, and direct-send call for all three methods. - DisplayMirrorViewModel lifecycle and managed-mode paths (feature/settings/.../DisplayMirrorViewModel.kt:74-125) have no tests. Disconnect, initialization reset, managed-mode transitions, blocked operations, allowed disable, stopMirroring, and onCleared remain unverified. Use fake flows and collaborators and assert reset calls, state changes, and forwarded arguments. - MirrorRectCompositor has only a one-chunk test that checks frame size (DisplayMirrorManagerImplTest.kt:284-291). The changed-panel rejection is tested, but valid multi-chunk pixel content, dirty-rectangle placement, and preservation of earlier canvas pixels are not tested. - DisplayMirrorManagerImpl still lacks boundary tests for negative and overflowing frame offsets, changed palette signatures, negative or inconsistent palette totals, and invalid palette region offsets. The reset test uses completed inputs (DisplayMirrorManagerImplTest.kt:217-225), so it does not prove that reset clears an in-progress mono frame, palette, or RGB565 rectangle. - MirrorFrameRenderer rendering paths (feature/settings/.../MirrorFrameRenderer.kt:140-223) have no pixel-level tests. MirrorColorResolverTest tests RGB565 conversion and a standalone region helper only. It does not test MONO_VLSB bit order, palette signature matching and fallback, region Y bounds, or little-endian RGB565 pixel reads. - DisplayMirror.kt and DpadRing.kt handlers (DisplayMirror.kt:259-369, DpadRing.kt:158-267) have no interaction tests. Touch scaling and clamping, swipe threshold and dominant-axis selection, Enter/Backspace/Tab forwarding, disabled controls, wedge boundaries, repeat timing, and select long-press behavior remain unverified. Add Compose tests with a test clock or extract the handlers for unit tests. - DebugScreen Mirror-tab entry and exit (Debug.kt:178-203, DisplayMirror.kt:133-134) have no test. Selecting the Mirror tab, preserving the existing tabs, and disposing the Mirror content to call stopMirroring remain unverified.

Resolution

Add focused AdminControllerImpl protobuf/destination tests; DisplayMirrorViewModel fake-flow lifecycle and managed-mode tests; valid multi-chunk RGB565 pixel and canvas-preservation tests; remaining malformed frame/palette boundary and partial-reset tests; renderer pixel tests; Compose or extracted-handler interaction tests; and DebugScreen tab-entry/exit lifecycle coverage.

Full details: Moved Code Diffed Against Its Original

Explanation

The visibility widening remains. Commit 4247323 moved private MirrorFrameImage from DisplayMirror.kt to internal MirrorFrameRenderer.kt:60. Commit e31e2d1 moved private input helpers and constants into internal declarations in DisplayMirrorUtils.kt:29-66,92; current callers and tests use these module-visible declarations. Commit 2323fee extracted private rectangle state into internal class MirrorRectCompositor; its handle, snapshot, reset, width, and height remain module-visible. No later commit restores the original file-private scope.

Resolution

Preserve the original private scope. Keep MirrorFrameImage and utility helpers in their caller file, or add private wrappers and test through the caller. Keep MirrorRectCompositor private or nested inside DisplayMirrorManagerImpl. If module-internal visibility is required for tests or reuse, document and intentionally approve that API-scope change instead of treating it as a mechanical move.

  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@jamesarich

Copy link
Copy Markdown
Collaborator Author

Cross-platform feature spec (client-agnostic wire contract, reassembly rules, input mapping, UX and device requirements): meshtastic/design#142

@codecov

codecov Bot commented Aug 31, 2026

Copy link
Copy Markdown

⚠️ JUnit XML file not found

The CLI was unable to find any JUnit XML files to upload.
For more help, visit our troubleshooting guide.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 4


ℹ️ Review info
⚙️ Run configuration
  • Configuration used: Repository: meshtastic/Meshtastic-Android/.coderabbit.yaml
  • Review profile: CHILL
  • Plan: Advanced
  • Run ID: 7cca8cc2-c670-41d2-9530-2371a6f445bb
📥 Commits

Reviewing files that changed from the base of the PR and between 77ae807 and 14b2f58.

📒 Files selected for processing: 25
  • .skills/compose-ui/strings-index.txt
  • core/data/src/commonMain/kotlin/org/meshtastic/core/data/manager/DisplayMirrorManagerImpl.kt
  • core/data/src/commonMain/kotlin/org/meshtastic/core/data/manager/FromRadioPacketHandlerImpl.kt
  • core/data/src/commonMain/kotlin/org/meshtastic/core/data/manager/MirrorRectCompositor.kt
  • core/data/src/commonTest/kotlin/org/meshtastic/core/data/manager/DisplayMirrorManagerImplTest.kt
  • core/data/src/commonTest/kotlin/org/meshtastic/core/data/manager/FromRadioPacketHandlerImplTest.kt
  • core/repository/src/commonMain/kotlin/org/meshtastic/core/repository/AdminController.kt
  • core/repository/src/commonMain/kotlin/org/meshtastic/core/repository/DisplayMirrorManager.kt
  • core/resources/src/commonMain/composeResources/drawable/ic_keyboard_arrow_left.xml
  • core/resources/src/commonMain/composeResources/drawable/ic_keyboard_arrow_right.xml
  • core/resources/src/commonMain/composeResources/values/strings.xml
  • core/service/src/commonMain/kotlin/org/meshtastic/core/service/AdminControllerImpl.kt
  • core/testing/src/commonMain/kotlin/org/meshtastic/core/testing/FakeRadioController.kt
  • core/ui/src/commonMain/kotlin/org/meshtastic/core/ui/icon/Navigation.kt
  • feature/settings/src/commonMain/kotlin/org/meshtastic/feature/settings/debugging/Debug.kt
  • feature/settings/src/commonMain/kotlin/org/meshtastic/feature/settings/debugging/DisplayMirror.kt
  • feature/settings/src/commonMain/kotlin/org/meshtastic/feature/settings/debugging/DisplayMirrorUtils.kt
  • feature/settings/src/commonMain/kotlin/org/meshtastic/feature/settings/debugging/DisplayMirrorViewModel.kt
  • feature/settings/src/commonMain/kotlin/org/meshtastic/feature/settings/debugging/DpadRing.kt
  • feature/settings/src/commonMain/kotlin/org/meshtastic/feature/settings/debugging/MirrorFrameRenderer.kt
  • feature/settings/src/commonMain/kotlin/org/meshtastic/feature/settings/radio/RadioConfig.kt
  • feature/settings/src/commonTest/kotlin/org/meshtastic/feature/settings/debugging/DisplayMirrorInputTest.kt
  • feature/settings/src/commonTest/kotlin/org/meshtastic/feature/settings/debugging/MirrorColorResolverTest.kt
  • gradle.properties
  • gradle/libs.versions.toml

Included review availability: This review used your included allowance. Your plan provides up to 4 included reviews per hour; 3 remain after this review.

Comment thread gradle.properties Resolved
@github-actions

Copy link
Copy Markdown
Contributor

protobufs 2.8.1.21-gf4c4ff9-SNAPSHOT → 2.8.1.22-gc858ad1-SNAPSHOT

Compare: meshtastic/protobufs@f4c4ff9...c858ad1

Merged upstream

Schema

meshtastic/admin.proto |  21 +++++
 meshtastic/mesh.proto  | 292 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
 2 files changed, 313 insertions(+)

Added lines: 34 fields, 9 enum values.

Settings strings (values/schema_strings.xml)

No settings string changes.

values/schema_strings.xml is regenerated by the scheduled-updates run this merge triggers on main.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1


Review info
Run configuration
Commits

Reviewing files that changed from the base of the PR and between 3a570ee and 71b6d91.

Files selected for processing: 8
  • .skills/compose-ui/strings-index.txt
  • core/data/src/commonMain/kotlin/org/meshtastic/core/data/manager/DisplayMirrorManagerImpl.kt
  • core/data/src/commonTest/kotlin/org/meshtastic/core/data/manager/DisplayMirrorManagerImplTest.kt
  • core/resources/src/commonMain/composeResources/values/strings.xml
  • core/service/src/commonMain/kotlin/org/meshtastic/core/service/AdminControllerImpl.kt
  • core/testing/src/commonMain/kotlin/org/meshtastic/core/testing/FakeRadioController.kt
  • feature/settings/src/commonMain/kotlin/org/meshtastic/feature/settings/debugging/DisplayMirror.kt
  • gradle/libs.versions.toml
Files skipped from review as they are similar to previous changes: 2
  • .skills/compose-ui/strings-index.txt
  • core/resources/src/commonMain/composeResources/values/strings.xml

Included review availability: This review used your included allowance. Your plan provides up to 4 included reviews per hour; 3 remain after this review.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1


Review info
Run configuration
Commits

Reviewing files that changed from the base of the PR and between 71b6d91 and 6d690aa.

Files selected for processing: 2
  • core/data/src/commonMain/kotlin/org/meshtastic/core/data/manager/DisplayMirrorManagerImpl.kt
  • core/data/src/commonTest/kotlin/org/meshtastic/core/data/manager/DisplayMirrorManagerImplTest.kt

Included review availability: This review used your included allowance. Your plan provides up to 4 included reviews per hour; 2 remain after this review.

Consumes the DisplayFrame PoC protobufs (2.99.0-screen-mirror-poc-SNAPSHOT
from mavenLocal, useMavenLocal committed on this branch): DisplayMirrorManager
reassembles FromRadio.display_frame chunks into MirrorFrame snapshots,
AdminController gains setDisplayMirror + sendInputEvent, and the Debug panel
gains a Mirror tab rendering the device's 1bpp framebuffer live with a D-pad
driving the firmware InputBroker. PoC: tab strings deliberately unlocalized.
Cap the mirror canvas at 512.dp (4x for a 128px OLED) instead of filling
the window width, fold the frame info into the toggle row, and pair
Down/Back so the controls fit without scrolling on desktop.
Manager: validate format (MONO_VLSB only) and geometry (width/height must
describe exactly total_size bytes) so malformed device input can't reach
aspectRatio and crash; unit tests for reassembly, drops, restarts.
AdminController: setDisplayMirror/sendInputEvent become non-suspend
immediate sends (teardown-safe, input never queues behind bulk admin) and
gain requestDisplayFrame. Mirror tab: render each frame once into a 1:1
ImageBitmap scaled with FilterQuality.None (kills the fractional-scale
seams), gate controls on connection state, reset the toggle on disconnect,
stop the stream on tab dispose and ViewModel clear, add a one-shot Refresh,
inject AdminController/ConnectionStateProvider instead of RadioController.
Dispatch: display_frame test coverage incl. revoked-session.
Populates the new DisplayInfo during the handshake — dimensions from the
live display device, MONO_VLSB frame format, PanelClass from the panel
defines (EINK/HUB75/TFT/LCD/OLED), has_touch from HAS_TOUCHSCREEN — so
clients can gate mirroring UI and adapt to the panel. Absent on
display-less builds. Regenerates protos (DisplayInfo, DisplayFrame rect
fields reserved for dirty-rect streaming).
Replace the flat button stack with the researched control surface:
cross-shaped 5-way cluster (56dp circular keys, 8dp gaps, Material
Symbols chevrons with contentDescriptions), directions auto-repeat on
hold (500ms then 10Hz — slow enough for the radio to render), OK
long-press sends SELECT_LONG, Back sits below per TV-remote convention.
The mirror itself becomes an input surface: click to capture the
keyboard (arrows / Enter / Esc, with a visible focus ring and hint) and
swipe in a cardinal direction to navigate. Consumes the new
DeviceMetadata.display: shows an e-ink hint steering toward one-shot
Refresh. Adds keyboard_arrow_left/right icons (Material Symbols rounded,
transposed from the existing up glyph).
DisplayMirrorManager reassembles FromRadio.display_palette chunks
(signature-keyed, defaults captured on the offset-0 chunk) into a
MirrorPalette; frames carry palette_signature and the renderer resolves
each pixel through the region table (last overlapping region wins,
matching firmware precedence) with RGB565 conversion, falling back to
monochrome when the referenced palette hasn't arrived. Renderer split
into MirrorFrameRenderer.kt.
jamesarich and others added 20 commits October 28, 2026 18:16
Controls sit beside the mirror when the content width allows (>=760dp,
BoxWithConstraints) and stack below it otherwise; tapping the mirror now
toggles keyboard capture instead of only acquiring it, with the hint text
naming the way out.
Circular D-pad ring replaces the cross cluster (wedge hit-testing with a
radial dead band, hold-to-repeat, per-wedge semantics, haptics), keyboard
capture becomes an explicit chip with tap-to-toggle, and taps on touch
devices forward real panel coordinates (long-press = SELECT, matching the
physical touch driver). Manager gains reset() (wired to ViewModel creation
and disconnect), a palette region cap, and first-chunk geometry pinning;
renderer uses bit-replicated RGB565 and run-length row drawing with the
pixel resolver extracted pure and tested; KeyUp of captured keys is
consumed; ViewModel moved to its own file; all strings localized; the
stray .bak test file is gone.
DisplayMirrorManager reassembles RGB565 rect chunks (sequence, geometry,
and panel-bound validation) and composites them into a persistent
little-endian RGB565 canvas emitted per completed rect; the renderer gains
a true-color branch using the same run-length row drawing. Mono frames are
untouched.
RGB565 frames identify LVGL/MUI devices, whose UIs read their own input
drivers rather than the InputBroker remote events inject into — so the
mirror hides the D-pad, keyboard, swipe, and touch affordances and says
so, until an input-injection path exists. Utility helpers and the input
codes move to DisplayMirrorUtils.kt.
device-ui now accepts injected input (virtual LVGL pointer + group-attached
keypad bridged from the InputBroker), so the view-only treatment goes away:
the D-pad navigates MUI widgets, taps forward panel coordinates, and the
keyboard types. The mirror_view_only string stays for the future
capability-flag path.
The mirror's keyboard capture only mapped navigation keys and silently
dropped everything else, so text entry did nothing: printable characters
never left the client even though the firmware already injects kb_char.
Characters now go out as ANYKEY with the code point, Backspace goes as a
character (so text fields delete rather than navigating back, which stays
Esc's job), and the handling moves into a Modifier.remoteKeyInput
extension.
Third adversarial review pass over the client side.

Rect reassembly moves out of DisplayMirrorManagerImpl into
MirrorRectCompositor: the manager was carrying two unrelated state
machines and 14 methods. Every rect field is now validated before
anything is allocated (total_size is device-controlled), geometry is
latched from the chunk at offset 0 so a later chunk cannot transpose
the blit, continuations must match the buffer that was actually
allocated rather than their own claim, and bounds use the subtraction
form because rect_x + rect_width overflows for hostile values. A rect
that omits width or height covers the full panel in that axis.

remoteKeyInput no longer swallows modified keys, so Cmd+Q, Ctrl+C and
Alt+Tab stay with the desktop instead of being forwarded to the device.

MirrorFrameRenderer tracks colorized runs as a raw Int with a sentinel
rather than a boxed Color?, and MirrorFrame.hashCode includes the
format so a mono and an RGB565 frame of the same size cannot collide.

Drops mirror_view_only, orphaned since MUI gained remote input.
The Debug Panel is the one Settings route deliberately exempt from the
`connected && !isManaged` gate every sibling carries, justified by a comment
saying it "reads app-local logs only". The Mirror tab put an admin write
(set_display_mirror) and arbitrary input injection on that route, so managed
mode - which exists to lock a device out of configuration - was bypassed
entirely.

Enforced in the ViewModel, because the route will not do it: arming,
one-shot frame requests, key, character and touch sends are all refused while
managed. Disarming still goes through, so a stream already running is torn
down when managed mode arrives mid-session.

The UI gates on `active = connected && !managed` rather than offering controls
that silently do nothing, and the RadioConfig comment now says what the route
actually guarantees, since its old wording is what let this through.

MirrorHeaderRow is split out of DisplayMirrorContent, which was at detekt's
LongMethod limit and could not absorb another line.
The mirror's key capture was a focusable box, so it could never raise the soft
keyboard. It now uses core/ui's RemoteKeyboardSink: the Keyboard chip raises
the IME, typed text reaches the device one code point per kb_char (emoji
included), Enter is OK, Esc is Back, Backspace and Tab are typed characters,
and Ctrl/Alt chords stay with the host. The scroll column pads for the IME so
the D-pad stays visible. Space now types a space instead of meaning OK.
A 65536x524288 chunk wrapped `width * pages` to 0, matched `total_size = 0`
and published an empty frame that crashed the renderer. The size and offset
checks now run in Long and refuse a negative offset.
The D-pad went inactive in managed mode, but taps, swipes and the Keyboard
chip stayed live while the view model dropped the input. `MirrorSurface` now
takes `enabled`: gestures come off, the chip disables and keyboard focus clears.
Type-resolved detekt on main flags a LaunchedEffect whose body never suspends
(UnnecessaryLaunchedEffect).
A continuation chunk could lower region_total and complete a short palette under a valid signature, or add more regions than the offset-0 chunk declared. Remember the total from the first chunk and reject continuations that change or overshoot it.
Only firmware with display mirroring fills DeviceMetadata.display, so a null one means set_display_mirror and get_display_frame_request would go unanswered. Gate arming and frame requests on it and tell the user why; keys predate mirroring and stay live. Also drop the em dashes from the mirror strings.
@jamesarich
jamesarich added this pull request to the merge queue Oct 29, 2026
Merged via the queue into main with commit 04b9ab6 Oct 29, 2026
13 of 14 checks passed
@jamesarich
jamesarich deleted the screen-mirror-poc branch October 29, 2026 00:40
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

enhancement New feature or request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant