Add opt-in native-host image-backed edge lifecycle by esaurez · Pull Request #395 · microsoft/nvx

Enrique Saurez (esaurez) · 2026-10-05T18:00:23Z

Summary

Add an opt-in nvxhost backend to the edge sandbox crate. The caller supplies OpenVMM, a compatible kernel/initramfs and read-only GPT image, an absolute native-library path, and an independently approved SHA-256. The loader verifies ABI v1 and the required scratchless-launch and same-descriptor peer-check exports before use.
Keep the existing direct-OpenVMM backend as the default. Store native sandbox state separately under <state_root>/nvxhost/, own the OpenVMM process and guest boot console, and implement provision/start/exec/guest logs/stop/deprovision through the native ttrpc channel.
Reject unsupported mounts, network configuration beyond deny-all, custom execution environments or working directories, piped stdin, and cancellation. Snapshot/restore and image pulling or per-request image selection are outside this profile. Errors and output-limit overflows are explicit.
Bound native-session teardown by the caller's operation deadline, preserve explicit zero as an unbounded command timeout, reject positive non-whole-second timeouts, validate both derived Unix socket paths, and leave native-only filenames untouched in direct-backend state.
Keep the guest boot console attached. OpenVMM serves one console client at a time and retains guest output while none is connected, so console listeners are keyed by the OpenVMM process identity, retired after a crash, and wait to take over a console held by another backend instance or process. Stop and deprovision report capture failures as consoleError metadata instead of failing.
Register guest images by content ID (sha256:<hex>). Registration hashes an image once, or records a digest that the caller's own policy verified; provision and start compare cheap file seals instead of hashing again and fail closed when a file changed. OpenVMM, the kernel, and the initramfs are hashed once per backend, optionally against approved digests. Hashing again before every start remains an opt-in diagnostic. register_image, images, verify_image, and unregister_image manage registrations; unregistering refuses while a provisioned sandbox uses the image.
Claim the OpenVMM log before a native launch. OpenVMM inherits the claim, so if the caller dies before recording OpenVMM's identity, the next operation waits for OpenVMM's endpoint and terminates it, or clears the interrupted launch once nothing holds the log, instead of leaving the sandbox unusable. The default backend's launch markers and recovery are unchanged.
Document a concrete example invocation, the separately supplied inputs, and the native-library trust requirement. The example neither fetches nor builds those inputs; --image-sha256 optionally requires the image digest.

Dependencies and review base

Requires nanvix/openvmm#112; this branch pins its scratchless OpenVMM commit 9524065522764c8b80b3a4d48ec7877ddc16090b.
The native library and minimal ttrpc guest are built separately and are not included in this public repository. A caller-supplied digest alone is not an authenticated installation: callers must approve and protect the installed asset independently.
This draft targets the temporary esaurez/baseline-before-nvx-86661676 branch at db5138e361e26c698eaff9d627b8668899a1437d, the first parent of merge 86661676. The newer OpenVMM pin rejects the available AMD validation hosts under an unrelated CPU-profile policy. Rebase onto dev after that policy supports the hosts; do not merge into the temporary base branch. An earlier replay of this branch onto dev passed the unit suites; it has not been refreshed for this head.

Validation (head `ac47d53`)

Windows: cargo test --features nvxhost,testing,async (132 unit tests plus every integration suite) and the default cargo test (101 unit tests); Clippy with -D warnings on all targets for both configurations; cargo +1.89 check (the declared minimum Rust version) for both.
Linux (Rust 1.98.1): the same Clippy and test commands pass (130 and 99 unit tests).
The opt-in WHP suite (11 tests, --release --test-threads=1) passes on an AMD WHP host: the earlier nine lifecycle and robustness tests plus image registration, reuse, fail-closed changes, unregistering, trusted digests, runtime-digest pinning, and content verification. Every test checks that no OpenVMM process outlives its sandbox.
Start calls spend 12-17 ms outside guest boot, down from about 0.8 s of hashing the 538 MB test image. Creating a backend takes 0.55-0.73 s when it registers the image and 46-60 ms when it reuses the registration; with content verification, a start spends 0.53-0.74 s hashing. The documented example completes in 1.2-1.3 s, including the first registration.
Faster starts let the killed-start test land between the launch marker and OpenVMM's recorded identity, which left the sandbox unusable; the log claim fixes this, and the test passed five further runs. Allowing write sharing in the Windows release check fails both new recovery tests.
The crash-and-restart test showed Windows updating the OpenVMM executable's change time when it caches the file hash in an extended attribute, failing an unchanged file closed. Windows seals therefore omit the change time, which writers can set anyway; Linux seals keep it.

Notes

Executions on one sandbox share its single control connection: four concurrent one-second commands complete in about 6 s.
A guest that writes enough console output stalls while no listener is attached, for example after the process that started it exits and before another process uses the sandbox.
Native sandbox state written by earlier revisions of this unmerged PR is not readable; stop and deprovision such sandboxes before upgrading. NvxHostConfig is now #[non_exhaustive]: construct it with NvxHostConfig::new and its builder methods.

The independent review's four medium-severity findings and its follow-up zero-timeout compatibility finding are addressed in e17cfae. adfa167 changes documentation only. 22cfcb1 fixes the boot-console listener and adds the robustness suite; an independent review found no significant issues. ac47d53 adds the image registry and the launch-log claim; its independent review raised the two compatibility points in the notes and nothing else.

Load a separately supplied, caller-approved native host library for scratchless image-backed provision/start/exec/stop/deprovision. Keep the direct backend as default, isolate persisted state, fence unsupported capabilities, and pin the RAM-overlay OpenVMM change. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Bound guest-session disposal by each operation deadline, preserve zero as an unbounded command timeout while rejecting sub-second precision, keep native-only files outside direct-backend cleanup, and validate both derived Unix socket paths before loading the library. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Show a concrete WHP invocation, the separately supplied artifacts and trust requirement, and the unverified MSHV path. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot

Copilot review overview

🟡 Changes recommended

Native execution currently has deadline, output-contract, and restart console-pump correctness issues.

Review effort: Balanced
Findings: 4

Open (4)

Use byte fields to preserve non-UTF-8 process output · New
Stale completed pump handle blocks VM console restart · New
Connection delay exceeds configured shutdown timeout · New
Enforce one-megabyte aggregate output limit · New

What changed in this PR

Adds an opt-in, image-backed nvxhost lifecycle while retaining direct OpenVMM as the default.

Changes:

Adds native-library ABI bindings and guest lifecycle implementation.
Adds isolated state, artifact verification, console capture, and tests.
Documents and demonstrates configuration and required private assets.

File	Description
`aci_edge_sandboxes/src/openvmm/native.rs`	Implements the native backend lifecycle.
`aci_edge_sandboxes/src/nvxhost.rs`	Adds checked native ABI bindings.
`aci_edge_sandboxes/src/openvmm/state.rs`	Adds native state and console artifacts.
`aci_edge_sandboxes/src/openvmm/config.rs`	Generalizes Unix socket validation.
`aci_edge_sandboxes/src/openvmm/mod.rs`	Exports and integrates the backend.
`aci_edge_sandboxes/src/openvmm/launch.rs`	Updates test records.
`aci_edge_sandboxes/src/lib.rs`	Registers the feature-gated module.
`aci_edge_sandboxes/src/client.rs`	Adds the client constructor.
`aci_edge_sandboxes/tests/nvxhost_guest.rs`	Adds an ignored WHP lifecycle test.
`aci_edge_sandboxes/examples/nvxhost_lifecycle.rs`	Adds a runnable lifecycle example.
`aci_edge_sandboxes/README.md`	Documents setup, trust, and limitations.
`aci_edge_sandboxes/Cargo.toml`	Adds the feature and dependencies.
`aci_edge_sandboxes/Cargo.lock`	Locks new dependencies.

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

+    #[prost(string, tag = "2")]
+    stdout: String,
+    #[prost(string, tag = "3")]
+    stderr: String,


+            .console_pumps
+            .lock()
+            .map_err(|_| Error::backend_error("the boot-console pump table is unavailable"))?;
+        if pumps.contains_key(sandbox_id) {


+            let remaining = deadline.saturating_duration_since(Instant::now());
+            let mut session = self.connect(&runtime, &capability, remaining)?;
+            let grace_period_milliseconds = i64::try_from(remaining.as_millis())
+                .map_err(|_| Error::backend_error("guest shutdown grace period is too long"))?;
+            let response = session.unary(
+                "Shutdown",
+                &ShutdownRequest {
+                    grace_period_milliseconds,
+                }
+                .encode_to_vec(),
+                Some(remaining),
+            );


+    let reply = ExecuteCommandResponse::decode(response.as_slice()).map_err(|error| {
+        Error::backend_error("the guest returned an invalid execution result").with_source(error)
+    })?;
+    let _ = io.stdout.write(reply.stdout.as_bytes());
+    let _ = io.stderr.write(reply.stderr.as_bytes());


OpenVMM serves one boot-console client at a time and retains guest output while none is connected. The native backend keyed its console listener by sandbox only, so a restart after an OpenVMM crash reused the finished listener of the crashed launch; with no listener, a verbose guest stalled until start timed out. A second backend instance that reattached to a running guest could not connect while the first one held the console, and stop then failed after the guest had already stopped. Key listeners by the OpenVMM process identity and retire stale ones before a launch. A listener that finds the console held elsewhere waits, without the start deadline, to take it over and ends quietly when OpenVMM exits; a reset after OpenVMM exits ends the log. Stop and deprovision report capture failures as consoleError metadata instead of failing. Extend the opt-in WHP suite with exec semantics, lifecycle errors and restarts, reattachment from a new backend, a crashed guest, a guest that outlives the process that started it, starts killed at several points, parallel sandboxes, and concurrent commands. List OpenVMM processes in a way that fails loudly, so the leak checks cannot pass vacuously. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Hash the image once when it is registered, or record a digest that the caller already verified, and refer to it by content ID. Provision and start compare cheap file seals instead of hashing again, and fail closed when a file changed. Hash OpenVMM, the kernel, and the initramfs once per backend, optionally against approved digests, and keep full re-hashing as an opt-in diagnostic. Starts now cost about the guest boot time. Windows seals omit the change time: writers can set it, and the system updates it when it caches a file hash in an extended attribute, which failed an unchanged OpenVMM executable closed during testing. Claim the OpenVMM log before a native launch. OpenVMM inherits the claim, so recovery can clear an interrupted start that recorded no process identity once nothing holds the log, instead of leaving the sandbox unusable. Markers of the default backend are unchanged. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Enrique Saurez (esaurez) · 2026-10-07T19:48:34Z

Superseded by #418, which carries this change on dev as part of the whole native-host edge backend, over nanvix/openvmm#121, and is validated end to end on WHP and MSHV. Closing.

Enrique Saurez (esaurez) mentioned this pull request Oct 5, 2026

microVM: opt in to read-only distro with RAM overlay upper nanvix/openvmm#112

Closed

Enrique Saurez (esaurez) force-pushed the esaurez/edge-nvxhost-pre-time-abi branch from e17cfae to c35937b Compare October 5, 2026 23:16

Enrique Saurez (esaurez) changed the base branch from esaurez/baseline-before-nvx-86661676 to dev October 5, 2026 23:17

Document opt-in native-host edge example inputs

adfa167

Show a concrete WHP invocation, the separately supplied artifacts and trust requirement, and the unverified MSHV path. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Enrique Saurez (esaurez) force-pushed the esaurez/edge-nvxhost-pre-time-abi branch from c35937b to adfa167 Compare October 6, 2026 00:31

Copilot AI balanced review requested due to automatic review settings October 6, 2026 00:31

Copilot started reviewing on behalf of Enrique Saurez (esaurez) October 6, 2026 00:32 View session

Enrique Saurez (esaurez) changed the base branch from dev to esaurez/baseline-before-nvx-86661676 October 6, 2026 00:32

Copilot AI reviewed Oct 6, 2026

View reviewed changes

Enrique Saurez (esaurez) and others added 2 commits October 5, 2026 20:07

This was referenced Oct 6, 2026

Map host paths and apply egress rules in the native-host backend #407

Closed

Add an opt-in agent backend over a native sandbox library #418

Merged

Enrique Saurez (esaurez) closed this Oct 7, 2026

Navigation Menu

Repository navigation

Add opt-in native-host image-backed edge lifecycle - #395

Add opt-in native-host image-backed edge lifecycle#395
Enrique Saurez (esaurez) wants to merge 5 commits into
esaurez/baseline-before-nvx-86661676from
esaurez/edge-nvxhost-pre-time-abi

Enrique Saurez (esaurez) commented Oct 5, 2026 •

edited

Loading

Uh oh!

Copilot AI left a comment

Uh oh!

Enrique Saurez (esaurez) commented Oct 7, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

Enrique Saurez (esaurez) commented Oct 5, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Dependencies and review base

Validation (head ac47d53)

Notes

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Copilot review overview

🟡 Changes recommended

Uh oh!

Enrique Saurez (esaurez) commented Oct 7, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Enrique Saurez (esaurez) commented Oct 5, 2026 •

edited

Loading

Validation (head `ac47d53`)