Skip to content
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
71 changes: 69 additions & 2 deletions aci_edge_sandboxes/Cargo.lock

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

9 changes: 9 additions & 0 deletions aci_edge_sandboxes/Cargo.toml
Original file line number Diff line number Diff line change
Expand Up @@ -23,11 +23,16 @@ openvmm = []
bundled = ["openvmm", "dep:sha2"]
# Tokio-based asynchronous wrappers around the synchronous core.
async = ["dep:tokio"]
# Loads an explicitly supplied native host library and boots an image-backed ttrpc guest.
nvxhost = ["openvmm", "dep:libloading", "dep:prost", "dep:sha2-runtime"]
# Test doubles: an in-memory `MockBackend` and the `aci-edge-sandboxes-fake-openvmm` binary.
testing = []

[dependencies]
getrandom = "0.4"
libloading = { version = "0.8", optional = true }
prost = { version = "0.13", optional = true }
sha2-runtime = { package = "sha2", version = "0.10", optional = true }
serde = { version = "1.0.200", features = ["derive"] }
serde_json = "1.0.120"
thiserror = "2"
Expand Down Expand Up @@ -65,6 +70,10 @@ doc = false
name = "lifecycle"
required-features = ["openvmm"]

[[example]]
name = "nvxhost_lifecycle"
required-features = ["nvxhost"]

[lints.rust]
missing_docs = "warn"
unsafe_op_in_unsafe_fn = "deny"
Expand Down
116 changes: 111 additions & 5 deletions aci_edge_sandboxes/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -3,12 +3,15 @@
`aci_edge_sandboxes` is the Rust interface to the ACI Edge Sandboxes lifecycle.
It exposes provision, start, exec, stop, and deprovision through a pluggable
backend. The default backend drives the `openvmm` binary directly; no daemon or
Python tooling is involved at runtime.
Python tooling is involved at runtime. The optional `nvxhost` backend uses a
separately supplied native host library and a different, image-backed guest.

Each sandbox is a microVM that runs the NVX guest's Alpine Linux userland
directly from its initramfs. There are no image layers, scratch disks, or
container namespaces; workloads run as a non-root user in the guest itself, and
guest state lives in memory until the sandbox stops.
By default, each sandbox is a microVM that runs the NVX guest's Alpine Linux
userland directly from its initramfs. That backend has no image layers, scratch
disks, or container namespaces; workloads run as a non-root user in the guest
itself, and guest state lives in memory until the sandbox stops. The opt-in
native backend instead runs commands against a caller-supplied, read-only GPT
image with a RAM overlay.

The Cargo package, library, and directory are named `aci_edge_sandboxes`.

Expand Down Expand Up @@ -95,6 +98,8 @@ envelope (`version`, `phase`, and `containment`) belongs to the caller.
- Backends in this crate:
- `openvmm::OpenVmmBackend` (feature `openvmm`, default) drives the `openvmm`
executable.
- `openvmm::NvxHostBackend` (feature `nvxhost`) drives OpenVMM using a
separately supplied native host library and image-backed guest.
- `testing::MockBackend` (feature `testing`) implements the state machine in
memory for consumers' unit tests.
- `AsyncAciEdgeSandbox` (feature `async`) wraps `AciEdgeSandbox` for Tokio. Lifecycle calls run on
Expand All @@ -104,6 +109,107 @@ To add a backend, implement `Backend`. Declare only the capabilities it can
enforce, and report state-machine violations with the codes listed in the
[error mapping](#error-mapping).

## Image-backed native host backend (opt-in)

Enable `nvxhost` to use `AciEdgeSandbox::nvxhost(NvxHostConfig)`. This leaves the default
direct-OpenVMM backend and its Alpine guest unchanged. The caller supplies OpenVMM, a
compatible kernel and static edge-agent initramfs, a prepared GPT image, the absolute path
to `nvxhost.dll` (`libnvxhost.so` on Linux), and the independently approved SHA-256 of that
library. The Rust crate does **not** build, download, or publish the private library.
`NvxHostBackend::new` verifies the file digest and ABI version and requires the additive
`nvx_build_ramfs_launch_arguments` and `nvx_session_connect_verified` exports. A missing,
wrong-version, or wrong-digest library fails rather than falling back.

The image is attached read-only in OpenVMM's distro block slot. The edge guest validates
its GPT and p2+ ext4 layers and uses a RAM-backed tmpfs upper/work for its overlay; it
does not interpret p1 OCI configuration or require a scratch disk. OpenVMM must support
the explicit `nvx_overlay_upper=ramfs` scratchless topology. On Windows, the backend
assigns `//./pipe/openvmm-microvm-<NAME>` control and boot endpoints. It retains the
OpenVMM process/state, sends the 32-byte capability through stdin, checks the serving
process on the connected pipe/socket, and owns graceful-or-forced stop and deprovision.
State is kept separately under `<state_root>/nvxhost/`; a running VM is never silently
converted to the direct backend. `NvxHostBackend::guest_logs` reads a bounded non-follow
guest-log snapshot before stop.

The backend hashes OpenVMM, the kernel, and the initramfs once, when it is created
(`NvxHostBackend::runtime_digests`); `NvxHostConfig::with_runtime_digests` requires approved
digests. It also registers the configured image under `<state_root>/nvxhost/images/` by its
content ID, `sha256:<hex>` (`ImageId`). Registration hashes the image once;
`ImageDigest::Expect` additionally requires a digest, and `ImageDigest::Trusted` records a digest
that the caller's own policy verified without reading the file. Later backends and processes
reuse a registration that the file still matches without reading it. Provision records the image
ID and the runtime digests. Start compares each file's seal (volume, file ID, length, and
last-write time, plus the change time on Linux) with the one taken when the file was hashed,
instead of hashing again, and fails with `backend_unavailable` if anything changed, so a start
costs about the guest's boot time. A seal detects replacement or modification, not a writer that
deliberately restores timestamps. On Windows, start also keeps writers out of the files until
OpenVMM has opened them. Register a changed image again with `register_image`: unchanged content
keeps its ID, so sandboxes that use it start again. `images` lists the registrations and whether
each file still matches, `verify_image` hashes one again as a diagnostic, and `unregister_image`
refuses while a provisioned sandbox uses the image. `with_content_verification(true)` hashes the
image and runtime files again before every start, also as a diagnostic.

Start claims the sandbox's OpenVMM log before launching, and OpenVMM inherits the claim. If the
caller dies before it records OpenVMM's identity, the next operation waits up to
`start_timeout` for that OpenVMM to open its endpoint and then terminates it; once no process
holds the log, nothing of the interrupted launch runs, so the sandbox is usable again.

As with the direct backend, executions on one sandbox share its single control connection: a
concurrent exec waits up to `control_timeout` for the running one to finish. The backend
copies the guest boot console to `NvxHostBackend::console_log_path` from the process that
started or last used the sandbox. OpenVMM serves one console client at a time and holds
guest output while none is connected, so a guest that writes enough console output stalls
until a listener reconnects: keep that process running, or use the sandbox from its
successor, whose listener waits to take the capture over. Start reports `bootMilliseconds`
and `guestBuildId`; stop reports `forced` and, after a failed graceful shutdown,
`gracefulError`. A capture failure never fails stop or deprovision: both report it as
`consoleError`.

This first backend supports provision/start/exec/stop/deprovision with shell commands or
argv and a fixed caller-provided image. Positive execution timeouts must be whole seconds,
matching the guest RPC's precision; finer-grained timeouts fail validation rather than
silently extending execution. It explicitly rejects host file mappings, network
configuration beyond deny-all, piped stdin, execution cancellation, custom working
directories and environments. Snapshot/restore, image selection, and richer guest
operations are not part of this profile. See
[`examples/nvxhost_lifecycle.rs`](examples/nvxhost_lifecycle.rs) for a run requiring
`--openvmm`, `--kernel`, `--initrd`, `--image`, `--host-library`, `--host-sha256`,
`--state-root`, `--hypervisor`, and a command after `--`; an optional `--image-sha256`
requires the image's digest when it is registered. The ignored
`tests/nvxhost_guest.rs` exercises an actual WHP guest when the corresponding
`NVXHOST_TEST_*` paths and approved DLL digest are set.

For example, from `aci_edge_sandboxes` on a Windows WHP host, set the following
paths to compatible, separately built artifacts and a caller-prepared GPT disk
with p2+ ext4 layers (not a container image reference). Use the **edge** guest
initramfs, not the default Alpine or standard container guest initramfs:

```powershell
$openvmm = 'C:\path\to\openvmm.exe'
$kernel = 'C:\path\to\vmlinux'
$initrd = 'C:\path\to\nvx-edge-initramfs.cpio.gz'
$image = 'C:\path\to\layers.gpt'
$hostLibrary = 'C:\path\to\nvxhost.dll'
$approvedHostSha256 = '<64 hexadecimal digits from an independent trust policy>'
$stateRoot = 'C:\nvx-edge-state'

cargo run --release --locked --features nvxhost --example nvxhost_lifecycle -- `
--openvmm $openvmm --kernel $kernel --initrd $initrd --image $image `
--host-library $hostLibrary --host-sha256 $approvedHostSha256 `
--state-root $stateRoot --hypervisor whp -- 'printf READY'
```

Build the native library and the static edge initramfs separately from their
matching private sources; this example neither fetches nor builds them. Use the
pinned OpenVMM, which includes the scratchless RAM-overlay topology, and a kernel
compatible with that OpenVMM and guest revision. On Linux, supply a matching
`libnvxhost.so` and OpenVMM build and select `--hypervisor mshv`; the Linux edge
lifecycle has not yet been verified end to end.

The caller must independently approve and protect the native asset. Checking a caller-supplied
digest does not make a writable path or a self-declared digest trustworthy; use this profile
only with an immutable, externally authorized library installation.

## OpenVMM backend

### Configuration
Expand Down
Loading