Skip to content

Add support for nvidia driver selection at boot time - #1041

Merged
vigh-m merged 3 commits into
bottlerocket-os:developfrom
vigh-m:nvidia-multi-driver/core-kit
Sep 18, 2026
Merged

vigh-m merged 3 commits into
bottlerocket-os:developfrom
vigh-m:nvidia-multi-driver/core-kit

Conversation

@vigh-m

@vigh-m vigh-m commented Sep 9, 2026

Copy link
Copy Markdown
Contributor

Description of changes:
Adds boot-time NVIDIA driver-branch selection for dual-branch images.

The kernel-kit change (bottlerocket-kernel-kit#536) makes the lts and pb driver branches co-installable and namespaces each branch's units as nvidia-<branch>-*

ghostdog match-nvidia-branch

  • Implement new match-nvidia-branch subcommand. It resolves the target branch from settings.kernel.drivers.nvidia.branch (added in bottlerocket-settings-sdk#154), falling back to instance-type detection when the setting is unset
  • writes a /run/nvidia/branch-<branch> marker file

os package wiring

  • New nvidia-select-branch subpackage requiring nvidia-lts, nvidia-pb, only installed when nvidia-dual-branch image feature is enabled (twoliter#747), and building and nvidia variant
  • Adds nvidia-select-branch.service to execute ghostdog match-nvidia-branch in a dual driver image

Service reordering

  • ecs-gpu-init.service and generate-cdi-specs.service now order After=drivers.target instead of the branch-specific load-*-kernel-modules.service units, since those unit names are now per-branch.

Tested with workload tests via the following:

Backward compatibility (namespaced kernel-kit + existing core-kit/Twoliter)

  • nvidia-r580 namespaced kernel-kit; boot on:
    • g4dn
    • g5
    • g6
    • g6f
    • g7
    • g7f
  • nvidia-r595 namespaced kernel-kit; boot on:
    • g4dn
    • g5
    • g6
    • g6f
    • g7
    • g7f

Image feature — happy path

  • Enable image feature; boot on:
    • g4dn
    • g5
    • g6
    • g6f
    • g7
    • g7f.xlarge

Terms of contribution:

By submitting this pull request, I agree that this contribution is dual-licensed under the terms of both the Apache License, version 2.0, and the MIT license.

@vigh-m
vigh-m force-pushed the nvidia-multi-driver/core-kit branch 2 times, most recently from 4637cf1 to ea035aa Compare September 11, 2026 21:59
@vigh-m vigh-m changed the title Nvidia multi driver/core kit Add support for nvidia driver selection at boot time Sep 11, 2026
@vigh-m
vigh-m force-pushed the nvidia-multi-driver/core-kit branch from ea035aa to 8fa54ff Compare September 11, 2026 22:42
Comment thread sources/ghostdog/src/nvidia.rs Outdated
@vigh-m
vigh-m force-pushed the nvidia-multi-driver/core-kit branch 4 times, most recently from 42589aa to a2aa8d4 Compare September 17, 2026 20:18
@vigh-m
vigh-m marked this pull request as ready for review September 17, 2026 20:19
Comment thread packages/os/nvidia-select-branch-ordering.conf
Comment thread packages/os/os.spec Outdated

@ginglis13 ginglis13 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

still working through the PR, just two initial nits

Comment thread sources/ghostdog/src/main.rs Outdated
Comment thread sources/ghostdog/src/nvidia.rs Outdated
Comment thread sources/ghostdog/src/main.rs Outdated
Comment thread sources/ghostdog/src/nvidia.rs
Comment thread sources/ghostdog/src/nvidia.rs
Comment thread sources/ghostdog/src/nvidia.rs
Comment thread packages/ecs-gpu-init/ecs-gpu-init.service
Comment thread packages/nvidia-container-toolkit/generate-cdi-specs.service
Comment thread packages/release/nvidia-driver-branch-deprecation Outdated
@vigh-m
vigh-m force-pushed the nvidia-multi-driver/core-kit branch from a2aa8d4 to de0d119 Compare September 18, 2026 22:42

@ginglis13 ginglis13 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, thanks for the testing deets

Add `match-nvidia-branch`, which resolves the active NVIDIA driver
branch (settings.kernel.drivers.nvidia.branch, else PCI device
detection) and writes a marker file.

Make `match-nvidia-driver` flavour matching branch-aware so co-installed
lts and pb drivers select the right units.

Signed-off-by: Vighnesh Maheshwari <vighmah@amazon.com>
Order ecs-gpu-init and generate-cdi-specs after drivers.target so they
wait for the GPU devices and kernel modules to load, on both multi- and
single-driver variants.

Signed-off-by: Vighnesh Maheshwari <vighmah@amazon.com>
On dual-branch images both NVIDIA driver branches (lts and pb) are
co-installed, and one is selected at boot via the services in this
subpackage

- nvidia-select-branch.service: a oneshot, ordered before
  drivers.target, that runs `ghostdog match-nvidia-branch` to pick the
  branch and write the marker file.
- nvidia-<branch>-.service.d/10-branch-gate.conf: a systemd prefix
  drop-in. The trailing dash means the drop-in applies to every unit
  whose name begins with nvidia-<branch>- (the overlay service and all
  per-branch module-load units), gating them on
  ConditionPathExists=/run/nvidia/branch-<branch> and ordering them
  after nvidia-select-branch.service.
- drivers.target drop-in ordering drivers.target after the selection
  oneshot.

The subpackage is pulled in only on nvidia/nvidia-fips variants built
with the nvidia-dual-branch image feature; single-driver images omit it
and run their units unconditionally.

Signed-off-by: Vighnesh Maheshwari <vighmah@amazon.com>
@vigh-m
vigh-m force-pushed the nvidia-multi-driver/core-kit branch from de0d119 to 03e9fd9 Compare September 18, 2026 23:05
@vigh-m
vigh-m merged commit c8c868d into bottlerocket-os:develop Sep 18, 2026
3 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants