Add opt-in native-host image-backed edge lifecycle by esaurez · Pull Request #418 · microsoft/nvx

Enrique Saurez (esaurez) · 2026-10-07T01:29:37Z

Replaces #395, which carried this change on a historical base that predates OpenVMM's time ABI and CPU profiles. This PR rebases it onto dev; see "Changes since #395".

Summary

Add an opt-in nvxhost backend to the edge sandbox crate. The caller supplies OpenVMM, a compatible kernel/initramfs and read-only GPT image, an absolute native-library path, and an independently approved SHA-256. The loader verifies ABI v1 and the required scratchless-launch and same-descriptor peer-check exports before use.
Keep the existing direct-OpenVMM backend as the default. Store native sandbox state separately under <state_root>/nvxhost/, own the OpenVMM process and guest boot console, and implement provision/start/exec/guest logs/stop/deprovision through the native ttrpc channel.
Reject unsupported mounts, network configuration beyond deny-all, custom execution environments or working directories, piped stdin, and cancellation. Snapshot/restore and image pulling or per-request image selection are outside this profile. Errors and output-limit overflows are explicit.
Bound native-session teardown by the caller's operation deadline, preserve explicit zero as an unbounded command timeout, reject positive non-whole-second timeouts, validate both derived Unix socket paths, and leave native-only filenames untouched in direct-backend state.
Keep the guest boot console attached. OpenVMM serves one console client at a time and retains guest output while none is connected, so console listeners are keyed by the OpenVMM process identity, retired after a crash, and wait to take over a console held by another backend instance or process. Stop and deprovision report capture failures as consoleError metadata instead of failing.
Register guest images by content ID (sha256:<hex>). Registration hashes an image once, or records a digest that the caller's own policy verified; provision and start compare cheap file seals instead of hashing again and fail closed when a file changed. OpenVMM, the kernel, and the initramfs are hashed once per backend, optionally against approved digests. Hashing again before every start remains an opt-in diagnostic. register_image, images, verify_image, and unregister_image manage registrations; unregistering refuses while a provisioned sandbox uses the image.
Claim the OpenVMM log before a native launch. OpenVMM inherits the claim, so if the caller dies before recording OpenVMM's identity, the next operation waits for OpenVMM's endpoint and terminates it, or clears the interrupted launch once nothing holds the log, instead of leaving the sandbox unusable. The default backend's launch markers and recovery are unchanged.
Document a concrete example invocation, the separately supplied inputs, and the native-library trust requirement. The example neither fetches nor builds those inputs; --image-sha256 optionally requires the image digest.

Dependencies

Requires microVM: opt in to read-only distro with RAM overlay upper nanvix/openvmm#121, the same scratchless RAM-overlay change as microVM: opt in to read-only distro with RAM overlay upper nanvix/openvmm#112, rebased onto the OpenVMM revision that dev pins. This branch pins 4744ee7ca52fa3c9265402db17f98f6c31ac3da7.
The native library and minimal ttrpc guest are built separately and are not included in this public repository. A caller-supplied digest alone is not an authenticated installation: callers must approve and protect the installed asset independently. The matching library no longer passes nolapic_timer to RAM-overlay guests, which must tick on their LAPIC timer under the time ABI.
The stack continues in Map host paths and apply egress rules in the native-host backend #419 (host paths and egress), Add a loopback proxy, forwarded ports, and exec environments to the native-host backend #420 (proxy, forwarded ports, and exec environments), Test that denied paths stay hidden through their aliases in the native-host backend #421 (denied-path aliases), and Let the native-host backend boot on a host CPU profile #422 (host CPU profiles).

Changes since #395

The state store uses dev's lock_and_load, which the native backend shares, instead of its own copy.
The nvxhost feature no longer renames its sha2 dependency. Cargo refused the renamed copy beside the bundled build dependency, so --all-features, which the crate's CI checks use, failed to build.
The example's documentation names the kernel that the time ABI requires, and says that start fails with backend_error on a host that no built-in CPU profile serves.

Validation

Each PR head in this stack passes cargo test --all-features and Clippy with -D warnings on all targets with --all-features, on Windows with Rust 1.93 (135, 137, 139, 139, and 140 unit tests).
At the top of the stack (5b0a315), on Windows and on Linux with Rust 1.93: the crate's CI checks, which are fmt, Clippy with --all-features and with --no-default-features, tests with --all-features and with default features, docs with -D warnings, cargo +1.89 check, and, on Linux, the macOS build check. The pinned library test passes against freshly built nvxhost.dll and libnvxhost.so.
Guest suite at the top of the stack, 20 tests, --release --test-threads=1, each checking that no OpenVMM process outlives its sandbox, with dev's Linux 6.18.38 kernel:
- Windows/WHP on an AMD EPYC 7763, which amd.milan.v1 serves: all pass, in about 100 s, and again with a host profile;
- Linux/MSHV on an AMD EPYC 9V74, which no built-in profile serves: all pass with a host profile, in about 82 s.

Notes

Executions on one sandbox share its single control connection: four concurrent one-second commands complete in about 6 s.
A guest that writes enough console output stalls while no listener is attached, for example after the process that started it exits and before another process uses the sandbox.
NvxHostConfig is #[non_exhaustive]: construct it with NvxHostConfig::new and its builder methods.

Load a separately supplied, caller-approved native host library for scratchless image-backed provision/start/exec/stop/deprovision. Keep the direct backend as default, isolate persisted state, fence unsupported capabilities, and pin the RAM-overlay OpenVMM change. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Bound guest-session disposal by each operation deadline, preserve zero as an unbounded command timeout while rejecting sub-second precision, keep native-only files outside direct-backend cleanup, and validate both derived Unix socket paths before loading the library. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Show a concrete WHP invocation, the separately supplied artifacts and trust requirement, and the unverified MSHV path. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

OpenVMM serves one boot-console client at a time and retains guest output while none is connected. The native backend keyed its console listener by sandbox only, so a restart after an OpenVMM crash reused the finished listener of the crashed launch; with no listener, a verbose guest stalled until start timed out. A second backend instance that reattached to a running guest could not connect while the first one held the console, and stop then failed after the guest had already stopped. Key listeners by the OpenVMM process identity and retire stale ones before a launch. A listener that finds the console held elsewhere waits, without the start deadline, to take it over and ends quietly when OpenVMM exits; a reset after OpenVMM exits ends the log. Stop and deprovision report capture failures as consoleError metadata instead of failing. Extend the opt-in WHP suite with exec semantics, lifecycle errors and restarts, reattachment from a new backend, a crashed guest, a guest that outlives the process that started it, starts killed at several points, parallel sandboxes, and concurrent commands. List OpenVMM processes in a way that fails loudly, so the leak checks cannot pass vacuously. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Hash the image once when it is registered, or record a digest that the caller already verified, and refer to it by content ID. Provision and start compare cheap file seals instead of hashing again, and fail closed when a file changed. Hash OpenVMM, the kernel, and the initramfs once per backend, optionally against approved digests, and keep full re-hashing as an opt-in diagnostic. Starts now cost about the guest boot time. Windows seals omit the change time: writers can set it, and the system updates it when it caches a file hash in an extended attribute, which failed an unchanged OpenVMM executable closed during testing. Claim the OpenVMM log before a native launch. OpenVMM inherits the claim, so recovery can clear an interrupted start that recorded no process identity once nothing holds the log, instead of leaving the sandbox unusable. Markers of the default backend are unchanged. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot

Copilot review overview

🟡 Changes recommended

Stop deadlines can be exceeded, binary output is not preserved, and failed example startup can leak OpenVMM.

Review effort: Balanced
Findings: 3

Open (3)

Failed start skips cleanup, leaving the VM running · New
Execution output incorrectly restricts raw bytes to UTF-8 · New
Stop timeout is not bounded after connect consumes time · New

What changed in this PR

Adds an opt-in image-backed nvxhost lifecycle backend while preserving direct OpenVMM as the default.

Changes:

Adds native-library bindings and ttrpc lifecycle operations.
Adds content-addressed image registration, artifact sealing, and crash recovery.
Adds documentation, an example, and native guest tests.

File	Description
`aci_edge_sandboxes/Cargo.toml`	Defines the feature, dependencies, and example.
`aci_edge_sandboxes/Cargo.lock`	Locks new dependencies.
`aci_edge_sandboxes/README.md`	Documents configuration, trust, and lifecycle behavior.
`aci_edge_sandboxes/examples/nvxhost_lifecycle.rs`	Demonstrates native lifecycle usage.
`aci_edge_sandboxes/src/client.rs`	Adds the client constructor.
`aci_edge_sandboxes/src/lib.rs`	Gates the native binding module.
`aci_edge_sandboxes/src/nvxhost.rs`	Implements the checked native ABI and ttrpc transport.
`aci_edge_sandboxes/src/openvmm/config.rs`	Generalizes Unix socket-path validation.
`aci_edge_sandboxes/src/openvmm/images.rs`	Implements image registration and file sealing.
`aci_edge_sandboxes/src/openvmm/launch.rs`	Updates state fixtures.
`aci_edge_sandboxes/src/openvmm/mod.rs`	Exports the backend and extends launch recovery.
`aci_edge_sandboxes/src/openvmm/native.rs`	Implements native lifecycle operations.
`aci_edge_sandboxes/src/openvmm/platform/linux.rs`	Adds Linux seals and launch-log claims.
`aci_edge_sandboxes/src/openvmm/platform/mod.rs`	Defines portable file seals.
`aci_edge_sandboxes/src/openvmm/platform/other.rs`	Adds unsupported-platform stubs.
`aci_edge_sandboxes/src/openvmm/platform/windows.rs`	Adds Windows seals and launch-log claims.
`aci_edge_sandboxes/src/openvmm/state.rs`	Adds backend-aware native state.
`aci_edge_sandboxes/tests/nvxhost_guest.rs`	Adds ignored guest lifecycle tests.
`openvmm`	Pins the required scratchless-overlay revision.

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

+    let stopped = succeeded.then(|| client.stop(&id));
+    let deprovisioned = if stopped.as_ref().is_none_or(Result::is_ok) {
+        Some(client.deprovision(&id))
+    } else {
+        None
+    };
+    let mut failures = Vec::new();
+    if let Err(error) = &executed {
+        failures.push(format!("guest lifecycle: {error}"));
+    }
+    if let Some(Err(error)) = &stopped {
+        failures.push(format!("stop: {error}"));
+    }
+    if let Some(Err(error)) = &deprovisioned {
+        failures.push(format!("deprovision: {error}"));
+    }


+    #[prost(string, tag = "2")]
+    stdout: String,
+    #[prost(string, tag = "3")]
+    stderr: String,


+            let remaining = deadline.saturating_duration_since(Instant::now());
+            let mut session = self.connect(&runtime, &capability, remaining)?;
+            let grace_period_milliseconds = i64::try_from(remaining.as_millis())
+                .map_err(|_| Error::backend_error("guest shutdown grace period is too long"))?;


Enrique Saurez (esaurez) and others added 5 commits October 6, 2026 18:02

Document opt-in native-host edge example inputs

1ec99d0

Show a concrete WHP invocation, the separately supplied artifacts and trust requirement, and the unverified MSHV path. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot AI balanced review requested due to automatic review settings October 7, 2026 01:29

Enrique Saurez (esaurez) mentioned this pull request Oct 7, 2026

Map host paths and apply egress rules in the native-host backend #419

Draft

Copilot started reviewing on behalf of Enrique Saurez (esaurez) October 7, 2026 01:30 View session

Enrique Saurez (esaurez) mentioned this pull request Oct 7, 2026

microVM: opt in to read-only distro with RAM overlay upper nanvix/openvmm#121

Draft

Copilot AI reviewed Oct 7, 2026

View reviewed changes

Navigation Menu

Repository navigation

Add opt-in native-host image-backed edge lifecycle - #418

Add opt-in native-host image-backed edge lifecycle#418
Enrique Saurez (esaurez) wants to merge 5 commits into
devfrom
esaurez/nvxhost-edge-backend

Enrique Saurez (esaurez) commented Oct 7, 2026 •

edited

Loading

Uh oh!

Copilot AI left a comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

Enrique Saurez (esaurez) commented Oct 7, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Dependencies

Changes since #395

Validation

Notes

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Copilot review overview

🟡 Changes recommended

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Enrique Saurez (esaurez) commented Oct 7, 2026 •

edited

Loading