Skip to content

openvmm: add an authenticated microVM host pause endpoint - #114

Draft
esaurez wants to merge 2 commits into
mainfrom
esaurez/microvm-vm-state-control
Draft

esaurez wants to merge 2 commits into
mainfrom
esaurez/microvm-vm-state-control

Conversation

@esaurez

@esaurez esaurez commented Oct 6, 2026 •

Copy link
Copy Markdown

Summary

Its base, #115 (AMD CPUs in CPU profiles), has merged as 07850d27a, so this PR now targets main directly with its two commits, 130b936ad and 647e635c3.

A microVM launched with a live authenticated control console (--microvm-control-console listen=... --microvm-control-auth-stdin) cannot use the management RPC, the console relay, or --paused, and its REPL is headless, so its host has no way to pause or resume it.

This adds --microvm-state-control listen=PATH, a separate local endpoint for host pause, resume, and run-state queries:

  • Bound and peer-checked like the control console. On Linux it is an AF_UNIX 0600 socket in an owned 0700 directory with an SO_PEERCRED UID check. On Windows it is a named pipe with a protected DACL and a token-SID check.
  • Authenticated with the existing control-console capability (stdin) within --microvm-control-auth-timeout-ms. A failed or stalled authentication, or a malformed request, closes the connection without a response. One host is served at a time, with a 60 s idle timeout.
  • Protocol v1 has the AUTHENTICATE, QUERY, PAUSE, and RESUME requests. Each response carries the run state (RUNNING, PAUSED, STOPPED, BUSY), a transitions counter, and the broker instance ID. The protocol is documented in Guide/src/reference/openvmm/management/state_control_protocol.md.
  • Served outside the VM's state units, so it answers while the VM is paused. It is never recorded in a snapshot.

Worker semantics

This adds VmRpc::MicrovmPause, VmRpc::MicrovmResume, and VmRpc::MicrovmRunState:

  • Pause stops the state units and holds guest monotonic time: VP 0's TSC (the time ABI capture anchor) and every VP's LAPIC state.
  • Resume sets the TSC and LAPIC state back with the time ABI's synchronized TSC set before any vCPU runs. VM time stops and starts with the state units.
  • Wall-clock time is not held. The CMOS RTC and the time sample keep following host UTC, and the guest steps its clock at its next wall-clock poll (the NVX discipline reads the time sample).
  • Pause and resume are idempotent and atomic in the worker. Both report BUSY during a snapshot boundary or post-restore gate.
  • Pause is rejected for periodic or TSC-deadline LAPIC timers, as capture is. The guest continues after a brief stall.
  • A guest reset while paused discards the held time but keeps the pause. A failed reset ends the pause.
  • TimeAbiBackend::set_synchronized_tsc is now also called at each host resume. Every instantiated VP is bound before the VM loads, so MSHV's VP-creation seal is unaffected.

Also

serial_socket: when an accepted client's identity cannot be resolved (for example, the client already exited), the Windows named-pipe backend now listens again. Previously it dropped the pipe instance and panicked on the next poll ("polled after completion"). This also affects the control console.

Testing

These results are for the branch rebased on #115 (07850d27a).

  • Windows (cargo +1.95):

    • Unit tests cover the protocol codec, session and authentication handling, status mapping, a real named-pipe endpoint session, CLI validation, the worker host-pause state, the snapshot-boundary filter, and the VP-set LAPIC fan-out. They pass: openvmm_core 103, openvmm_entry 191, vmm_core 65, virt 36, openvmm_defs 18, and serial_socket 4.
    • A deterministic named-pipe regression test fails without the serial_socket fix.
    • clippy --all-targets -- -D warnings and xtask fmt are clean.
  • Linux: cargo check and clippy --all-targets --target x86_64-unknown-linux-gnu are clean.

  • Linux: the unit tests pass with Rust 1.98.1 in NVX's pinned build container on an MSHV host, including the real AF_UNIX endpoint session: openvmm_core 103, openvmm_entry 223, vmm_core 65, virt 36, openvmm_defs 18, cpu_profile 111, and serial_socket 1.

  • Runtime: NVX's new pause-resume scenario (test-microvm: pause and resume a microVM through OpenVMM state control microsoft/nvx#405) holds a managed 8-vCPU VM paused for 30 s and then twice for 2 s. It passes, together with managed-lifecycle, on two backends:

    • MSHV on an AMD EPYC 9V74 (Zen 4) host. No pinned profile covers Zen 4 yet, so a test-only wrapper added --cpu-profile host; test-microvm has no option for it.
    • WHP on an AMD EPYC 7763 (Milan) host, on amd.milan.v1.
    Backend Paused Guest minus host running time RCU stalls Wall clock behind after resume Repaired after
    MSHV 34.0 s 0.007 s 0 34.4 s 8.1 s, one 34.01 s step
    WHP 34.1 s 0.003 s 0 34.9 s 8.2 s, one 34.10 s step

    On MSHV, OpenVMM used 1.01 CPU seconds per second while the guest was busy and none while it was paused. On both backends, hosts that did not authenticate or presented the wrong capability got no response.

  • Not run: KVM and Intel hosts. NVX CI runs the scenario on its Intel KVM, MSHV, and WHP runners and on the debug kernel once Cherry-pick: vmbus_server: call unstick_channels to mitigate the lost synic issue microsoft/openvmm#405 leaves draft.

esaurez and others added 2 commits October 6, 2026 08:42
A microVM that has a live authenticated control console cannot use the
management RPC, the console relay, or --paused, and its REPL is
headless, so its host had no way to pause or resume it.

Add --microvm-state-control listen=PATH, a separate local endpoint that
is bound and peer-checked like the control console. A host
authenticates with the control-console capability within the
control-console authentication timeout, then sends QUERY, PAUSE, and
RESUME requests. The endpoint serves one host at a time, answers while
the VM is paused, and is never recorded in a snapshot. The Guide
documents protocol version 1.

The VM worker gains MicrovmPause, MicrovmResume, and MicrovmRunState. A
host pause stops the state units and holds guest monotonic time: once
the vCPUs stop, it takes VP 0's TSC and every LAPIC, and resume sets
them back with the time ABI's synchronized TSC set before any vCPU
runs. Wall-clock time is not held; the CMOS RTC keeps following host
UTC. A pause is busy during a snapshot boundary or post-restore gate,
and is rejected for periodic or TSC-deadline LAPIC timers, as capture
is. A guest reset while paused keeps the pause but discards the held
time; a failed reset ends the pause.

Also make the Windows named-pipe serial backend listen again when an
accepted client's identity cannot be resolved, instead of dropping the
pipe and panicking on the next poll.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
The NVX guest's wall-clock discipline reads the time ABI's time sample,
not the CMOS RTC. Both follow host UTC across a host pause; say so, and
that the guest steps its clock from such a source at its next poll.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@esaurez
esaurez force-pushed the esaurez/microvm-vm-state-control branch from d3669f3 to 647e635 Compare October 6, 2026 15:55
@esaurez
esaurez changed the base branch from main to fix/amd-cpu-profiles-396 October 6, 2026 15:55
Base automatically changed from fix/amd-cpu-profiles-396 to main October 6, 2026 17:25
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant