[whoisaldo]

Fixes #183

What changes

The iPad draws the PC's pointer itself, from small messages sent apart from the video. On a still desktop, pointer motion no longer needs a video frame, in Mirror and on the Second screen.

Negotiation. The iPad sets HELLO2 feature bit 5 (FEATURE_DRAWS_POINTER). The host answers with host capability bit 6 (HOSTCAP_POINTER). Only when both are set does the host leave the pointer out of the video and send it separately. Each side ignores the other's unknown bit, so an older iPad or an older host gets the previous behaviour. Pointer messages start once the iPad has answered, as audio does. The pointer stays in the video, and the positions sent say hidden, until the iPad's report names the pointer image it holds whole. Then capture stops drawing it, once per session.

Wire. Two new control types, each with a body version byte, append-only like the rest:

• POINTER_SHAPE (0x1C): the pointer image, in chunks of up to 1,358 bytes. It is premultiplied BGRA plus, when any pixel inverts the picture under it, a 1-bit invert mask. Each image has an id, a hotspot, and its original Windows type.
• POINTER_STATE (0x1D): position (top-left of the image, in video pixels, may be negative), visibility, image id, and the host's read time. The newest wins, by msg_seq.
• RECEIVER_REPORT gains an appended pointer_shape_id, naming the image the iPad holds complete. When a report sent at least 250 ms after an image went still names another id, the host sends the image again. The image also goes at every session start and IDR.
• An image counts as sent once the link has taken every chunk. When the USB tunnel's control queue is full, the chunks resume 5 ms later where it stopped.

Host.

• Desktop Duplication shapes are converted once per change:
  • Color: premultiplied.
  • Monochrome: exact. Black, white, clear and invert.
  • Masked color: mask-0 pixels are opaque. Mask-1 pixels XOR the picture with their color: black changes nothing, so it is clear, and every other color is drawn as invert. That is exact for white, the color Windows uses.
• Positions are relative to the captured output, where the video has always drawn the pointer.
• The transport sends at most one position per frame budget, the newest one, on a fixed grid of budgets. It waits for the next budget on the transport's high-resolution deadline thread. The last position goes once more 100 ms later and again with every heartbeat.
• While the connected iPad draws the pointer, capture leaves it out of the video, and a pointer-only update is released without a readback or a new frame. The idle keepalive and keyframe answers still run on those updates, as on a still desktop.
• Each capture starts with the pointer hidden, so a position from the output captured before (the mirror a second screen starts from) is not drawn on the new one.
• When a session starts or ends with a different answer, the cached picture is rebuilt from the staging texture so it has, or lacks, the pointer.

Second screen.

• The virtual display driver gives Windows a hardware cursor only when its settings file turns one on. The file the host writes had no <cursor> block, so Windows drew the pointer into the second screen's image.
• With "match the iPad's resolution" on (the default), the host replaces that file before each Second screen start, as before. It now also writes the driver's own cursor block: hardware cursor on, 128x128, alpha and XOR support. With the option off, the host changes only the graphics card in the existing file, as before.
• The driver reads that file only when it starts. An Enable that starts it (the driver read as off before it and running after it) records that start in vdd-cursor.json in the app data folder:
  • the boot id;
  • the device's last arrival time (DEVPKEY_Device_LastArrivalDate). On the reference PC it changed at each driver start and stayed the same while the driver was off;
  • the SHA-256 of the settings file the driver started with;
  • the cursor that file sets: the hardware cursor switch, its maximum size, alpha support and the XOR level. They are read as the driver reads them: only true turns a switch on, numbers as std::stoi reads them, and comments don't count. A character or entity reference inside a cursor setting makes it unknown.
• Each Second screen capture compares the running driver's boot and arrival with that record:
  • Same start: the record decides. With a hardware cursor, each pointer image is checked against the caps that start used (for example the user's own 64 or 256): one larger than them stays in the picture, and so do all images without a hardware cursor. Windows then draws the pointer into the picture; the host draws none of its own, and tells the iPad it is hidden.
  • Any other start, or no record: the host can't tell. This covers a quick reconnect, the app restarted with the driver left on (when the record is missing), and the driver restarted by another program. Then the host does as 0.3.4: it draws the pointer Desktop Duplication reports into the video, and sends no image or shown position to the iPad. If the iPad was already drawing the pointer from the mirror before, it gets only hidden positions.
• The host also can't tell (and does as 0.3.4) when:
  • the driver's settings folder (VDDPATH under its registry key) is set to anything but C:\VirtualDisplayDriver, ignoring case and a trailing backslash;
  • a cursor value under that key can override the file;
  • a read of any of those values fails for a reason other than "not there";
  • the driver couldn't be read as off before the Enable (only a driver seen off and then running started with the file just written);
  • the file turns the cursor on without full support for every pointer type;
  • the installed driver's version is older than 24.10.27 (the first release that reads <cursor>) or doesn't read as a version. That version is the installer's, so it isn't taken to mean the running driver has no hardware cursor.

iPad.

• PointerOverlay reassembles images. It ignores chunks of an image older than the one held or assembling (comparison that survives the id wrapping), and keeps the newest position.
• MetalRenderer draws the image over the video in the same pass: alpha "over", then a second pass that inverts the picture where the mask is set (blend (1 - dst)·src + dst·(1 - src)). Both passes are clipped to the picture, so a pointer past its edge stays out of Fit's bars.
• The rectangle comes from ContentRectMapper, the same rectangle touches are normalized against, in Fit and Fill.
• A pointer change redraws the held frame.

Compatibility

iPad	PC	Pointer
this	this	drawn by the iPad, apart from the video (Mirror; Second screen when the host knows the running driver has its hardware cursor; otherwise in the video, as before)
this	older	in the video, as before (no HOSTCAP_POINTER; the iPad never draws)
older	this	in the video, as before (no FEATURE_DRAWS_POINTER; no pointer messages sent)
older	older	unchanged

Measurements

Windows 11 reference PC with an RTX 5080, H.264 NVENC, 120 fps target.

• A fake client over Wi-Fi counted frames and pointer states.
• The pointer was moved with SendInput for 60 s over a still window, at 1,200 px/s:
  • steady: about 610 updates/s;
  • bursty: 4 updates 1 ms apart, then 30–60 ms with none;
  • ~125 Hz with jitter.
• "Update → out" runs from each SendInput to the moment that position leaves the host step that hands it to the client. Before this change, that step is the encoder taking the frame, and encoding and sending follow. With it, it is the pointer state being sent.

Mirror, 3440x1440:

Input	Build	Pointer positions at the client /s (median / p10 / min)	Video frames /s while only the pointer moves	Update → out, mean / p90 / p99 ms
steady	before	116 / 115 / 113 (as frames)	116	5.6 / 9.2 / 10.4 (to the encoder)
steady	this, iPad draws	120 / 120 / 113	1	3.5 / 6.9 / 8.1
steady	this, older client	116 / 115 / 113 (as frames)	116	5.7 / 9.2 / 10.4 (to the encoder)
bursty	before	56.5 / 54 / 53 (as frames)	56.5	4.2 / 9.0 / 10.2 (to the encoder)
bursty	this, iPad draws	53.5 / 49 / 43	1	1.9 / 6.0 / 8.0
~125 Hz	before	116 / 102 / 88 (as frames; 69 lost on Wi-Fi)	116	5.5 / 8.8 / 10.0 (to the encoder)
~125 Hz	this, iPad draws	120 / 119 / 112	1	2.6 / 6.2 / 7.9

Second screen, the virtual display at 2732x2048, with the driver's hardware cursor in this build:

Input	Build	Pointer positions at the client /s (median / p10 / min)	Video frames /s while only the pointer moves	Update → out, mean / p90 / p99 ms
steady	before	116 / 115 / 91 (as frames; Windows draws the pointer into the image)	116	not measurable: the reported position is stale
steady	this, iPad draws	120 / 120 / 118	1	3.8 / 6.6 / 8.1
steady	this, older client	116 / 88 / 62 (as frames; 98 lost on Wi-Fi)	116	5.6 / 8.6 / 10.3 (to the encoder)
bursty	before	57 / 55 / 53 (as frames)	57	not measurable, as above
bursty	this, iPad draws	56 / 48.5 / 48	1	1.5 / 5.6 / 7.8

With a 160 px pointer on the second screen (larger than the hardware cursor takes), Windows drew it into the picture: about 114 frames/s while it moved, and every position sent said hidden.

Second screen with the driver-start record (this build, three cold starts, a client that draws the pointer):

• Each time the host recorded the driver's start and capture used it: hardware cursor, 128 px.
• While the pointer moved: 120 pointer positions/s at the client and 1.3 video frames/s.
• The decoded frames have no pointer at the client's position.

App relaunch with the driver left on:

• When the app is ended, its failure reporter turns the virtual display off, and the app also turns it off when it starts. So a relaunch starts the driver again and records that start.
• With both of those skipped, the record matched the running driver, and the host kept it.
• The second screen then did not attach: the attach check waits for the display to arrive after the Enable. The host mirrored the main screen. This change doesn't touch that check.

While only the pointer moves and the iPad draws it, the encoder takes only the idle keepalive (about 1.3 frames/s).

Tests

• Wire (Rust and Swift, shared golden vectors): new vectors hello2_draws_pointer, hello_ack_pointer, pointer_shape, pointer_state, pointer_hidden, receiver_report_pointer. Every truncation and the bad ids, sizes, lengths and versions are rejected on both sides. An older parser reads the extended report unchanged. The base's vectors are unchanged.

• Host unit tests:

  • Shape conversion for all three Windows types, including monochrome and masked-color XOR.
  • Coalescing under bursts and under a 125 Hz-like mouse with jitter (that test fails with a budget counted from the last send).
  • The budget's end is within one budget and never earlier than the last.
  • Repeat and refresh, hiding, and when images are resent.
  • Keepalive and keyframe answers during pointer-only updates.
  • Capability gating in the session and in client_draws_pointer.
  • The second screen's pointer placement against the running driver's caps (128, and a user's 64 and 256), without a hardware cursor, and when the driver can't be told. Compositing never draws a pointer Windows drew, and draws the reported one where the driver can't be told.
  • The driver-start record: Enable from off, AlreadyDone with a matching record, AlreadyDone after the arrival changed, a relaunch reading the record from disk, a reboot, and a damaged record. No record after an Enable when the driver was running before it or couldn't be read.
  • The driver's state from a scan of root devices: any device that couldn't be located or whose hardware IDs couldn't be read makes it unreadable, so no record is kept.
  • When the registry leaves the file in charge: no VDDPATH or the default folder in any case and with a trailing backslash; any other folder, a failed read, or any cursor value present or unreadable is unknown.
  • An old or unreadable driver version is unknown, not "no hardware cursor".
  • The driver settings file read as the driver reads it: switches, std::stoi numbers, missing sizes, alpha and XOR levels, white space and comments; references inside a cursor setting are unknown.
  • A withheld pointer: no messages to an iPad that was never shown one, and only hidden positions to one that was.
  • Desktop offsets: a tap on the drawn hotspot maps to the hotspot's desktop pixel, within the input mapping's own 1 px rounding.

• iPad unit tests:

  • Reassembly in any order with repeats.
  • An older image's chunks arriving after a newer one completed, also across the id wrap.
  • Newest position wins across the counter wrap.
  • Mask expansion.
  • The overlay rectangle against ContentRectMapper and the video quad in Fit and Fill.
  • The clip to the picture for a pointer crossing the edge in Fit.
  • Session and capability gating in ControlChannel.

• Host end to end (e2e_synthetic, real pipeline over UDP, synthetic still picture with the pointer moving at 60 fps, 3 s): a client that draws the pointer got 181 shown pointer states and 3 video frames; hidden states came only before the first shown one (25), and any hidden state after it fails the test. An older client on the same host got 179 frames, and any pointer message to it fails the test.

• iPad end to end (scripts/e2e_pointer.sh, Simulator, 10 s):

  App	Pointer positions applied/s	Pointer moves drawn/s	Video frames decoded/s
  draws the pointer	57.5	52.5	1.3 (idle keepalive)
  EM_DRAWS_POINTER=0 (as older apps)	0	0	59.9