[2/3] vfio: PCI device passthrough — config, BARs, DMA, KVM binding - #93
Draft
maxandrews wants to merge 13 commits into
Draft
[2/3] vfio: PCI device passthrough — config, BARs, DMA, KVM binding#93maxandrews wants to merge 13 commits into
maxandrews wants to merge 13 commits into
Conversation
…tures) CUDA_PASSTHROUGH Phase 1 step 0. QEMU-virt/Cloud-Hypervisor-compatible windows, clear of the GIC and virtio-MMIO bands, valid for both boot modes. ECAM 0x3000_0000 (1M), 32-bit MMIO 0x1000_0000 (512M), 64-bit MMIO 0x40_0000_0000 (256G). Foundation for the PCI transport + VFIO passthrough.
Phase 1 (steps 1-3 of CUDA_PASSTHROUGH_IMPL.md): a new src/devices/src/pci/ module ported from Cloud Hypervisor's pci crate, behind a new devices 'pci' feature (+ a 'vfio' feature that depends on it). Trimmed of CH's serde snapshot state, vm-migration hooks, live MsixConfig back-reference and vm-allocator BAR hooks; byteorder replaced with native from_le_bytes. aarch64 is ECAM-only so PciConfigIo is dropped, and PciConfigMmio is adapted to libkrun's BusDevice (no cross-vCPU Barrier return). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EqKBqYeUZbH9cmvs7G2qb7
…bus) - fdt/aarch64: emit a pci-host-ecam-generic node (ECAM reg + 32/64-bit ranges) under the 'pci' feature. - device_manager/mmio: register_pci() inserts PciConfigMmio at PCIE_ECAM_BASE. - builder: attach_pci_root() creates the PciBus (host bridge at 00:00.0) with a no-op DeviceRelocation and registers the ECAM window; called during aarch64 device setup. New StartMicrovmError::RegisterPciDevice. - Cargo: 'pci' and 'vfio' feature fan-out across libkrun/vmm/devices crates. Phase 1 (steps 1-3): guest should now enumerate an empty PCI bus. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EqKBqYeUZbH9cmvs7G2qb7
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EqKBqYeUZbH9cmvs7G2qb7
msb builds msb_krun (src/krun), not the libkrun cdylib crate, so the pci/vfio feature fan-out must exist here too. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EqKBqYeUZbH9cmvs7G2qb7
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EqKBqYeUZbH9cmvs7G2qb7
Steps 4-5 of CUDA_PASSTHROUGH_IMPL: a hand-rolled VFIO type1v2 wrapper (container/group/device open, region info, config pread/pwrite, DMA-map + mmap helpers) and a VfioPciDevice that presents a passed-through device on the guest PCI bus — config space passes through to the physical device, BARs are emulated and allocated from the guest PCIe MMIO windows. builder: attach_vfio_device opens devices named in KRUN_VFIO_PCI and adds them to the bus. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EqKBqYeUZbH9cmvs7G2qb7
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EqKBqYeUZbH9cmvs7G2qb7
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EqKBqYeUZbH9cmvs7G2qb7
Vm::register_mmio_memslot + Vmm delegate; attach_vfio_device mmaps each mmappable BAR from the device fd and registers it as a KVM memslot at the allocated guest BAR address, so guest BAR accesses reach the hardware. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EqKBqYeUZbH9cmvs7G2qb7
attach_vfio_device now identity-maps all guest RAM into the device IOMMU (VFIO_IOMMU_MAP_DMA) so the GPU can DMA, and binds the group to KVM via KVM_DEV_VFIO/GROUP_ADD. Expose VfioDevice::container()/group_as_raw_fd(). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EqKBqYeUZbH9cmvs7G2qb7
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EqKBqYeUZbH9cmvs7G2qb7
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EqKBqYeUZbH9cmvs7G2qb7
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
[2/3] vfio: PCI device passthrough — config, BARs, DMA, KVM binding
Assigns a real PCI device (selected by BDF via
KRUN_VFIO_PCI, comma list)into an aarch64 guest: the device appears on the emulated PCIe bus with correct
BARs, its memory is mmap'd into the guest, guest RAM is DMA-mapped into the
device IOMMU, and the VFIO group is bound to KVM. No interrupts yet — MSI-X
lands in [3/3].
What's in it
vfio-ioctlscrate):container/group/device, region info, config
pread/pwrite,map_dma,mmap_region.VfioPciDevice: PciDevice— config-space passthrough with emulatedBARs (guest sees guest-side BAR addresses/sizes; vendor/device/class/caps
pass through).
mmap→ KVM memslots (register_mmio_memslot).VFIO_IOMMU_MAP_DMA).KVM_DEV_TYPE_VFIO+KVM_DEV_VFIO_GROUP_ADD).vfiocargo feature (impliespci), off by default.Commits (8)
vfio-bindings dep·VfioPciDevice (config + BAR alloc) + builder attach·BAR
mmap→memslot (step 6) · IOMMU DMA-map + KVM-VFIO bind (steps 7-8) · plus afew small warning/cleanup commits.
Testing
lspci/sysfs in the guest show the device (e.g. an NVIDIA T4G10de:1eb4) withcorrect 32/64-bit BARs; the microVM boots and runs (BAR mmap + DMA + KVM bind all
succeed). Interrupt delivery is added and end-to-end-verified in [3/3].
Stacking — please review after #92
This is part 2 of 3 and is stacked on #92. GitHub can't base a PR on a
branch from a fork, so this PR targets
krundirectly; until #92 merges its 5commits also appear in this diff — the incremental change here is the 8 VFIO
commits on top of #92 (
pr/pcie-ecam-host-bridge..pr/vfio-passthrough).🤖 Generated with Claude Code