fix(aiperf-bench): move runtime to NVIDIA distroless python

[mchmarny]

Summary

Move the aiperf-bench runtime stage from python:3.13-slim to nvcr.io/nvidia/distroless/python:3.13-v4.0.8 (pinned by digest), and replace the /bin/sh -c benchmark invocation with a shell-free Python entrypoint since the distroless base ships no shell.

Motivation / Context

Requested move to the NVIDIA distroless Python base for the AIPerf benchmark image. The swap is not drop-in: the base has no shell, no apt, no pip, and no useradd, which breaks both the builder stage and the benchmark Job's runtime contract. This PR does the work those constraints require and re-validates the vulnerability posture end to end.

Fixes: N/A
Related: N/A

Type of Change

• Bug fix (non-breaking change that fixes an issue)
• New feature (non-breaking change that adds functionality)
• Breaking change (fix or feature that would cause existing functionality to change)
• Documentation update
• Refactoring (no functional changes)
• Build/CI/tooling

Component(s) Affected

• Validator (pkg/validator) — validators/performance only
• Other: aiperf-bench image, .openvex.json, .grype.yaml, RELEASING.md

Implementation Notes

Why only the runtime stage moved. The distroless image has no shell, apt, or pip, so it cannot host RUN steps. The builder stage stays on python:3.13-slim and still compiles the venv; only the final stage changed. Both bases put CPython 3.13 at /usr/local/bin, so the venv's interpreter symlinks and compiled extensions (pyzmq, uvloop, crick) resolve unchanged.

Replacing /bin/sh -c. buildAIPerfJob chained aiperf, echo, and cat in a shell to produce the sentinel framing parseAIPerfOutput consumes. That framing moved to validators/performance/aiperf_entrypoint.py (stdlib only), invoked in exec form. Division of responsibility:

• Go owns every benchmark flag. The wrapper only prepends aiperf profile and appends the framed result. This keeps flag correctness — including the --model flag that aiperf 0.11.0 requires — asserted in Go tests rather than in an untested Python file.
• The model is now a discrete argv element instead of a "$AICR_MODEL" shell expansion. This is strictly safer: with no shell there is nothing to re-scan a metacharacter-bearing value. Verified by running the image with model $(touch /tmp/pwned) and confirming the payload arrives verbatim and does not execute.
• AICR_MODEL is still set on the container, now informational, so kubectl describe pod shows what a run benchmarked.

Non-root user. RUN useradd cannot run, so the image adopts the base's built-in nvs user (uid 1000) instead of minting uid 10001. Still non-root; verified uid 1000 at runtime.

Vulnerability posture. The distroless base ships two Debian packages older than python:3.13-slim:

	python:3.13-slim (before)	distroless (after, with VEX)
Critical	0	0
High	2	2
Medium	14	15
Low	4	4
VEX-suppressed	11	32

• The 2 remaining Highs are the pre-existing CPython CVE-2026-11940 / CVE-2026-11972. The swap does not fix them — both bases ship CPython 3.13.14.
• libexpat1 2.7.1-2 contributes 21 findings (4 High). Suppressed in .openvex.json on two independently verified grounds: (1) nothing in the image links it — reading ELF DT_NEEDED for every binary under /usr/local/lib, /usr/lib, /lib, /opt/venv/lib, /usr/local/bin, /usr/bin returns zero references to libexpat; (2) CPython routes xml.parsers.expat through a pyexpat whose DT_NEEDED is ['libc.so.6'] only, with expat statically bundled at 2.8.1. aiperf itself has zero XML imports across its 682 .py files. Suppression is scoped to the unused system library; for the 13 CVEs fixed only in 2.8.2 the impact statement explicitly states the bundled 2.8.1 is not itself patched rather than overclaiming.
• liblzma5 5.8.1-1 (CVE-2026-34743, Medium) is deliberately left unsuppressed. _lzma.so genuinely links it, so the evidence is weaker than for expat, and python:3.13-slim already ships the fixed 5.8.1-1+deb13u1. Leaving it visible preserves the signal that the base image is stale. This is the one net-new finding in this PR.

Recommended follow-up: file upstream with the NVIDIA distroless team to refresh the Debian layer — 3.13-v4.0.8 was built 2026-06-11 and both stale packages have fixes available in Debian trixie.

VEX housekeeping. Dropped 11 aicr-gate statements (GO-2026-5005/5006/5013/5014/5015/5017/5018/5019/5020/5021/5023) confirmed entirely absent from the current scan after a chainsaw bump — verified individually, not inferred from a count delta. Document bumped to version 10 with a refreshed timestamp and tooling note. The remaining aicr-gate statements still apply (count-neutral).

Note for reviewers: two unsuppressed Highs remain on aicr-gate — GO-2026-5970 (x/text) and GO-2026-5942 (x/net). Those want a dependency bump, not a VEX entry, and are out of scope here.

Testing

make qualify   # exit 0

• make qualify passes clean on the final tree.
• golangci-lint run -c .golangci.yaml ./validators/performance/... → 0 issues.
• go test -race ./validators/performance/ passes; coverage 56.4% (unchanged — validators/ is excluded from the project floor per [Bug]: Validator unit tests are excluded from standard CI #1752).
• Image built for both linux/amd64 and linux/arm64; aiperf --version → 0.11.0 in the distroless runtime.
• Wrapper exercised end-to-end inside the real distroless image with a stub aiperf:
  • happy path emits sentinel / JSON / sentinel and parses correctly;
  • the newline guard is covered (aiperf writes the export without a trailing newline, so without it the closing sentinel would share a line with the JSON);
  • non-zero aiperf exit propagates verbatim (exit 7) with no sentinel emitted;
  • missing result file is a hard error rather than an empty sentinel block;
  • missing required env fails closed;
  • runs as uid 1000.
• New regression test ties the wrapper's exact output shape to parseAIPerfOutput.
• Grype re-scan used the CI-pinned v0.110.0 with the same --only-fixed --vex .openvex.json -c .grype.yaml flags, against an image tagged so the PURL resolves to pkg:oci/aicr-aiperf-bench; all 4 new High suppressions confirmed landing under appliedIgnoreRules with namespace = "vex".

Risk Assessment

• Low — Isolated change, well-tested, easy to revert
• Medium — Touches multiple components or has broader impact
• High — Breaking change, affects critical paths, or complex rollout

Rollout notes: The benchmark Job's Command/Args shape changes, so a released aicr binary must run against a matching aiperf-bench image. The validator resolves the inner image from the same tag as the outer validator (resolveAiperfImage / AICR_VALIDATOR_IMAGE_TAG), so they move together in normal use. A pinned older aiperf-bench image combined with a new binary would fail at pod start on the missing /opt/aicr/aiperf_entrypoint.py. Revert is a straight rollback of the Dockerfile and Go change.

Checklist

• Tests pass locally (make test with -race)
• Linter passes (make lint)
• I did not skip/disable tests to make CI green
• I added/updated tests for new functionality
• I updated docs if user-facing behavior changed
• Changes follow existing patterns in the codebase
• Commits are cryptographically signed (git commit -S)

[github-actions]

Coverage Report ✅

Metric	Value
Coverage	81.6%
Threshold	80%
Status	Pass

Coverage Badge

![Coverage](https://img.shields.io/badge/coverage-81.6%25-brightgreen)

Coverage unchanged by this PR.

[mchmarny]

All four review items addressed in 4517c5d (force-push; make qualify exit 0 on the result).

Item	Resolution
.openvex.json tooling count said "four" libexpat1 statements	Corrected to 21 (4 High, 16 Medium, 1 Low) — verified against the file
aiperf_entrypoint.py uncaught FileNotFoundError	Catches OSError, exits 127 (what the /bin/sh -c predecessor returned for command-not-found) rather than 1
Duplicated shell-indicator check in tests	Extracted to assertNoShellIndicators
.grype.yaml stale Reviewed: date (outside-diff comment)	Refreshed 2026-06-01 → 2026-08-03, and added libexpat1 to the suppressed-package list in the same comment block

The force-push also corrected a mistake of mine: an over-eager squash had absorbed upstream #1987 (the codeql-action bump) into this branch. The branch is now a single commit parented directly on origin/main touching exactly the 7 intended files.

Re-verified after the changes: wrapper exit paths in the rebuilt distroless image (missing binary → 127, happy path → 0 with correct sentinel framing, aiperf failure → 7), golangci-lint 0 issues, go test -race ./validators/performance/ green.

[njhensley]

📋 Multi-Persona Review — ✅ Approve with comments

Method: 4 independent persona reviewers (Correctness · Security/Supply-chain · Domain & Architecture · Test-coverage) → an adversarial senior meta-reviewer that re-derived every finding from the resolved code at 1612e034.
Legend: 🔴 Blocker · 🟠 Major · 🟡 Minor · 🔵 Nitpick

This is a well-executed, honestly-documented base-image swap. The meta-reviewer independently reproduced and upheld the load-bearing claims:

• Injection surface is genuinely eliminated — subprocess.run([bin, "profile", *argv], check=False), shell=False, no os.system; the model is a discrete argv element handed to execvp, so a metacharacter-bearing name is inert with no quoting.
• Exit-code parity with the removed /bin/sh -c holds — 127 exec-not-found, 128+N signal (OOMKilled still reports 137), verbatim non-zero, fail-closed on a missing result file. Benchmark-flag equivalence is exact.
• The VEX work is rigorous, not hand-wavy — the 13 libexpat CVEs fixed only in 2.8.2 explicitly state the statically-bundled 2.8.1 is not itself patched and fall back to the independent unreachability argument, rather than a false "already patched." No suppression coverage is actually lost by the 11 dropped GO- statements — each has a retained GHSA twin for the same CVE.
• Non-root posture preserved (uid 1000 nvs), digest pin correct, RELEASING.md updated.

No blocker or major finding survived adjudication. Two 🟡 items are worth addressing (a py_compile/test gate for the Python wrapper, and a factual correction to the .openvex.json tooling note); the rest is polish. Inline comments follow.

Confirmed non-issues (examined, cleared)

• Injection surface — genuinely eliminated.
• libexpat suppressions — not over-claimed; the 2.8.2-only CVEs correctly disclaim "already patched."
• Dropped GO- statements — no coverage lost; retained GHSA twins cover the same CVEs.
• Exit-code parity + fail-closed on missing result file — holds.
• Non-root (uid 1000 nvs), digest pin, benchmark-flag equivalence — all sound.

Tier summary

🔴 Blocker	🟠 Major	🟡 Minor	🔵 Nitpick	Recommendation
0	0	3	6	Approve with comments

_{Reviewed with a multi-persona + adversarial meta-reviewer workflow. Findings are anchored to the diff at 1612e034.}

[mchmarny]

All 9 review items addressed in 86fc21f6 (force-push; make qualify exit 0 on the result).

Item	Resolution
🟡 Python wrapper had no test or compile gate	py_compile gate in the builder stage + aiperf_entrypoint_test.go, which executes the real wrapper against a stub aiperf (exit contract: 7 / 137 / 127 / 1 / 2) and feeds its actual stdout to parseAIPerfOutput
🟡 VEX note blamed a chainsaw bump that never happened	Corrected — chainsaw is unchanged at v0.2.15; the 11 dropped statements are GO-ID duplicates of retained GHSA ones (verified 1:1). The commit message carried the same claim and was corrected too
🟡 Dockerfile↔Go coupling test was tautological	TestAIPerfEntrypointPathsMatchDockerfile reads the Dockerfile and asserts the COPY destination and venv layout against the Go constants
🔵 Newline-guard comment claimed a parse failure	Claim was wrong (strings.Index + TrimSpace); comment corrected and a glued-sentinel case added to prove it
🔵 Parse-shape test hand-encoded the wrapper output	Now derived from a real aiperf_entrypoint.py run
🔵 Artifact dir double-sourced	Assertion that the --output-artifact-dir argv equals the AICR_AIPERF_ARTIFACT_DIR env
🔵 GO-namespace fragility	Addressed via the corrected tooling note; the GO-ID duplicates stay dropped (details in-thread)
🔵 Rename AICR_MODEL	Declined — the name is shared with the model-cache Job, where it is read (model_cache.go:347)
🔵 Digest-pin the builder FROM	Declined here — all five Dockerfiles pair a floating builder with a digest-pinned runtime; worth doing as one sweep

Every new assertion was mutation-checked: removing the newline guard, the 128+N signal mapping, the py_compile line, the runtime COPY destination, or the argv/env artifact-dir agreement each fails the suite.

Thanks for the multi-persona pass — the VEX audit-trail catch in particular was the one that would have quietly cost us the retained GHSA suppressions later.

[coderabbitai]

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@validators/performance/inference_perf_test.go`:
- Around line 981-983: Correct the t.Errorf message in the envAIPerfModel
assertion to state that the variable is required for pod visibility or kubectl
describe output, not for the wrapper to pass --model. Keep the validation
condition unchanged.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: ASSERTIVE

Plan: Enterprise

Run ID: fc13a6b1-8022-4983-9d76-ea490b844036

📥 Commits

Reviewing files that changed from the base of the PR and between 1612e03 and 86fc21f.

📒 Files selected for processing (8)

• .grype.yaml
• .openvex.json
• RELEASING.md
• validators/performance/aiperf-bench.Dockerfile
• validators/performance/aiperf_entrypoint.py
• validators/performance/aiperf_entrypoint_test.go
• validators/performance/inference_perf_constraint.go
• validators/performance/inference_perf_test.go

[coderabbitai]

📐 Maintainability & Code Quality | 🟡 Minor | ⚡ Quick win

Correct the assertion message: the wrapper does not read AICR_MODEL.

The message states the env var exists "so the wrapper can pass --model". aiperf_entrypoint.py never reads it. inference_perf_constraint.go lines 149-152 and the wrapper docstring lines 29-31 both state the model reaches aiperf as a discrete --model argv element, and that the env var exists only for kubectl describe pod visibility. The wrong message invites a future change that makes the wrapper depend on the env var.

📝 Proposed fix

 	if env[envAIPerfModel] == "" {
-		t.Errorf("%s must be set so the wrapper can pass --model", envAIPerfModel)
+		t.Errorf("%s must be set for operator visibility via `kubectl describe pod`", envAIPerfModel)
 	}

📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change

	if env[envAIPerfModel] == "" {
	t.Errorf("%s must be set so the wrapper can pass --model", envAIPerfModel)
	}
	if env[envAIPerfModel] == "" {
	t.Errorf("%s must be set for operator visibility via `kubectl describe pod`", envAIPerfModel)
	}

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@validators/performance/inference_perf_test.go` around lines 981 - 983,
Correct the t.Errorf message in the envAIPerfModel assertion to state that the
variable is required for pod visibility or kubectl describe output, not for the
wrapper to pass --model. Keep the validation condition unchanged.

[njhensley]

🔁 Re-Review — ✅ Approve

Delta reviewed: 1612e034 → 86fc21f6 (fix commit). Disposition of all 9 prior findings against the resolved code + an adversarial pass on the fix surface. Delta is test + docs only (4 files). Validated at the fix head: go test -race PASS, golangci-lint 0 issues, py_compile OK, .openvex.json valid.

Thanks for the thorough turnaround — the fixes aren't cosmetic: the new +416-line aiperf_entrypoint_test.go executes the real Python wrapper end-to-end with a stub aiperf, and the Dockerfile now byte-compiles the wrapper in the builder and ships the exact file it accepted.

Prior-feedback status

Finding	Tier	Disposition
F1 — Python wrapper untested / no compile gate	🟡	✔️ Addressed — py_compile gate in builder + test asserts exit 7/137/127/1/2 on the real wrapper
F2 — openvex note "chainsaw bump" factually wrong	🟡	✔️ Addressed — reframed as GHSA-alias dedup; states chainsaw unchanged at v0.2.15
F5 — Dockerfile↔Go path test tautological	🟡	✔️ Addressed — TestAIPerfEntrypointPathsMatchDockerfile reads the real Dockerfile (residual precision gap noted inline as B1)
F3 — comment claims newline guard needed to parse	🔵	✔️ Addressed — comment corrected + new glued-sentinel parse case
F4 — parse test hand-encodes wrapper output	🔵	✔️ Addressed — TestAIPerfEntrypointFramingFeedsParser feeds real wrapper stdout to parseAIPerfOutput
F6 — artifact dir double-sourced, no guard	🔵	✔️ Addressed — argv --output-artifact-dir == env assertion added
F7 — "confirmed absent from scan" un-reproducible	🔵	◐ Mitigated — reworded note is now concrete (GHSA-alias mapping); inherent VEX-repro difficulty remains
F8 — AICR_MODEL dead-but-set naming	🔵	✖️ Not addressed — out of delta (optional nit)
F9 — builder floating tag vs digest-pinned runtime	🔵	✖️ Not addressed — optional nit

Adversarially cleared (no new issue)

• CI gate correct — requirePython3 fatals when CI is set; GitHub Actions always exports CI=true and ubuntu-latest ships python3, so the Python contract tests genuinely execute in CI and never silent-skip.
• No __pycache__/.pyc shipped — runtime COPY --from=builder names the exact .py; shipped bytes byte-identical.
• SIGKILL→137 deterministic; no goroutine/race/timeout fragility; no vacuous assertions.

One new low-severity follow-up nit (B1, inline): the Dockerfile-parser test is stage-blind. Not a blocker.

Verdict

🔴 Blocker	🟠 Major	🟡 Minor (open)	🔵 Nitpick (new)	Recommendation
0	0	0	1 (B1)	Approve

_{Re-reviewed with a multi-persona + adversarial meta-reviewer workflow, dispositioned against the fix commit at 86fc21f6.}

[njhensley]

🔵 Nitpick — Dockerfile-parser test is stage-blind; runtime COPYs aren't required to carry --from=builder

TestAIPerfEntrypointPathsMatchDockerfile flattens all Dockerfile lines ignoring FROM stage boundaries, and the copiesVenv/copiesToRuntime matchers check only src == dst, not the --from=builder flag. A refactor that dropped --from=builder (copying from the build context instead of the builder stage) or misplaced a COPY across stages would still satisfy the guard.

Blast radius: Narrow — the common breakages this test targets (renamed dest, deleted COPY, removed py_compile, moved venv path) are all still caught, and dropping --from=builder would usually break docker build outright, so this is a low-exploitability precision gap rather than a real hole.

Fix: Optional: assert the two runtime COPY lines contain --from=builder, and/or track the current FROM stage while scanning. Follow-up nit, not a merge blocker.