Skip to content

fix(runsc): backport upstream lock-free and bounded inventory - #12

Merged
rcyw merged 5 commits into
akernel-dev:developfrom
hky1999:hky1999/runsc-state-query-20261025
Nov 6, 2026
Merged

rcyw merged 5 commits into
akernel-dev:developfrom
hky1999:hky1999/runsc-state-query-20261025

Conversation

@hky1999

@hky1999 hky1999 commented Oct 24, 2026 •

Copy link
Copy Markdown
Collaborator

Slow restore can hold the lifecycle state-file flock or leave the Sentry control RPC temporarily unresponsive, blocking node-wide inventory on the older AKernel baseline. This PR now imports Google's fixes, rather than maintaining a second implementation.

Upstream changes

Base: akernel-dev/gvisor:develop at 63de4896444eb140221c1284fa621394666224af.

  • 9fa3d71a600e1b89a6c64f27f206ee942fe41262: atomic state-file publication.
  • 728f546aff7d3005c2a8d373f68c5d7e963d629f: shared pidfd process-liveness helper, needed by the identity changes.
  • f76c600d6ea58f6f43ffec982da69e69bd7b3f58: parallel list checks with a shared five-second wait; Created root containers use process liveness without querying a still-booting Sentry.
  • bab1123c2e369d601bba5ae7f9bed24b96a77596: read-only lock-free loading, write guards and pidfd process identity.

All retain the original authors and cherry-pick -x provenance. Conflict resolutions preserve upstream algorithms: map sentryapi.RuntimeStateStopped to the baseline boot type; retain older adjacent gofer fields and metric-registration code, applying read-only loading at the load sites that exist on this baseline. The upstream pidfs identity fallback remains usable on older kernels without pidfs.

Removed the previous custom Loader TryLock, cancellable state-RPC APIs, 16-query worker pool, and one-second RPC budget. These are not implicitly necessary just because they existed in the earlier candidate. Prior comments and timing results for a300cef8 / 2cfd3f66 describe the superseded implementation, not this head.

Contract and validation

Upstream list waits up to five seconds for live status, then loads saved metadata without the lifecycle flock. Sandboxd's inventory poll has a ten-second timeout. A direct state RPC can still block during restore; that diagnostic is distinct from bounded CLI inventory. Saved status does not imply workload readiness. Local filesystem stalls are not covered by the RPC wait budget. Upstream's log-and-skip behavior for unreadable records is retained.

Tests include atomic concurrent state-file readers and failed-publication preservation; upstream booting/unresponsive process, read-only guard and PID identity regressions; plus real FUSE-gated restore success, failure, Sentry exit, paused Sentry, and multiple simultaneous restores. The CLI acceptance deadline is explicitly seven seconds, rather than the earlier candidate's two seconds.

Measured validation

Head: e809115b58cbb5163d31c70f3aef21e76197a142. Four upstream commits plus one test/CI commit; the earlier custom implementation has been removed. The amd64 complete bundle used below is stamped 9151bec1ae93 (SHA256 d7578ed9e3ef8e46c43914f228436266529f3fafd741d116114c0ecbcd7d6532). Its entire source tree matches the final head except .github/workflows/akernel-pr.yml; subsequent changes only fixed privileged CI execution, then consolidated test commits. This is source-equivalent runtime validation, not a claim that the archive was rebuilt with the final Git stamp.

  • Local compilation and eight focused suites passed: config, boot, filestore snapshot, serialization, specutils, atomic state-file readers, upstream command lifecycle tests, and metricserver. Atomic/identity suites also passed ten repetitions each.
  • Real FUSE-gated restores on Systrap and KVM passed success, read EIO, Sentry exit, paused Sentry, and eight simultaneous restores. Each platform recorded 33 successful gated inventory calls; maxima 66.6 / 62.0 ms. Paused live-Sentry queries intentionally exercise upstream's five-second saved-state fallback (5.012–5.026 s).
  • Public-SDK isolation acceptance on 13 workers (Linux 5.10, direct XFS, durable checkpoint sync and reflink enabled): 300 × 2 rounds and 1000 × 2 rounds all passed. Each source is deleted before restore; original PID/start time, nonce, 64 MiB touched random memory, 32 MiB writable file, open-file and established internal stream state are checked. No guest Python coordination is needed.
Requested concurrency Successful restores / proofs SDK checkpoint p50 / p95 SDK restore-return p50 / p95 Full payload proof p50 / p95
300 600 / 600 3.80 / 4.38 s 0.55 / 0.74 s 3.97 / 8.71 s
1000 2000 / 2000 9.83 / 88.92 s 3.03 / 13.14 s 24.95 / 47.36 s

Full proof is timed from restore submission and includes the bounded proof queue; it is not bare restore execution. Clients use an isolated API via kubectl port-forward. Dispatch spans are 0.13–0.14 s for 300-way restore and 0.48–0.64 s for 1000-way restore. The 300 cohort is balanced across workers; the 1000 cohort has 50 on each of 12 workers and 400 on one larger worker. The latter hits cgroup page-cache reclaim pressure (no OOM) and must not be presented as a balanced production capacity/performance result.

During restore through full proof: runsc CLI inventory 120/120 and 666/666 successful (max 0.072 / 0.781 s); Sandboxd List, health and available-runtime probes 118/118 and 675/675 successful per probe. runsc remained available; no periodic reconciliation timeout/unavailable log appeared in either complete cohort.

Separate residual finding: across the entire 1000 lifecycle, Sandboxd List RPC timed out five times at its ten-second probe deadline, all during the first checkpoint on the 400-sandbox worker, not during restore. A 90-byte Sandboxd status write spent 66.403 s in fsync in the same interval. Sandboxd performs this write inside statusStorage.UpdateSync's write lock while List needs Status.Get's read lock. This evidence identifies a separate Sandboxd state-publication/I/O issue; this PR does not change it and does not claim all lifecycle inventory APIs are timeout-free. The runsc CLI remained responsive and health/runtime availability stayed successful even then.

Final-head amd64/arm64 bundle, regression, gated-restore and amd64 race CI: run 42028254289 passed. Both architectures passed full-bundle builds, focused regressions and real gated-restore acceptance; amd64 atomic-metadata and process-identity race suites passed all three repetitions. The workflow retains acceptance and race-log artifacts.

Isolation and cleanup

No production deployment was changed; final comparisons of DaemonSet, Deployment, StatefulSet, ConfigMap and Service specs/data found no changes. The test used its own namespace, control plane, Redis, worker runtime directories and scoped SDK credentials. Both SDK catalogs and all 13 runsc inventories were empty at completion. All isolated Pods/namespace were removed, the 13 owned host directories emptied, port-forward/monitors stopped, and local temporary build mounts and ephemeral private credentials removed. Evidence and the complete candidate bundle are retained locally. No release is published by this PR; use the normal complete-bundle CI release process after review/merge.

@hky1999

hky1999 commented Nov 5, 2026

Copy link
Copy Markdown
Collaborator Author

Keeping this PR open as an independent runtime follow-up after consolidating the older Sandboxd/AKernel drafts.

The currently tested HK1019 framework pins the complete release-20261012.0-akernel.3 runsc bundle. Its release tag resolves to 63de489. At that exact source, Loader.containerRuntimeState still takes l.mu.Lock(), ContainerRuntimeState does not propagate the transient busy error, and TestContainerRuntimeStateDoesNotWaitForLoader is absent:

func (l *Loader) containerRuntimeState(cid string) ContainerRuntimeState {

The current release therefore does not contain this PR's non-blocking state-query change. Merged inclusionAI/sandboxd#98 retains the Sandboxd-side inventory waiting/admission behavior, which is not a replacement for this Sentry/Loader fix. This PR remains separately tracked and needs alignment with the current runtime baseline, preserving its newer HasNonExitingTasks behavior. This deployment/source audit and PR consolidation do not change the running bundle or the main thread's active pressure test.

@hky1999
hky1999 force-pushed the hky1999/runsc-state-query-20261025 branch from 0cb5583 to ed7c414 Compare November 5, 2026 03:34
@hky1999
hky1999 marked this pull request as ready for review November 5, 2026 03:41
@hky1999

hky1999 commented Nov 5, 2026

Copy link
Copy Markdown
Collaborator Author

Added the real slow-restore acceptance runner and recorded results in 01bf36d: instructions/results.

The complete amd64 bundle from ed7c414 was exercised in isolated PID/mount namespaces, on both Systrap and KVM. Each platform covered success (same original workload nonce and advancing counter), checkpoint-read EIO, Sentry SIGKILL during restore, and subsequent inventory state convergence. All of those transition/convergence checks passed and final inventories were empty. Local config_test, boot_test and filestore_snapshot_test passed.

The strict end-to-end inventory acceptance did NOT pass. Real state RPCs returned the expected transient busy error in 0.49–0.76 ms, but all 18 full runsc list queries during gated restore exceeded 2 seconds. Their captured kernel stacks show locks_lock_inode_wait → __x64_sys_flock. Container.startImpl holds the state-file lock through restore; list → container.Load waits there before reaching the repaired RPC.

I updated the PR description to separate the successful RPC fix from this remaining metadata-lock blocker. The test deliberately retains normal CLI locking and exits nonzero; it does not bypass the lock or drop a busy container from inventory. The new PR CI run is running; that workflow covers bundle/unit checks, not this manually executed strict FUSE acceptance.

No production deployment was changed and no bundle was published.

@hky1999 hky1999 changed the title fix(runsc): keep runtime state queries responsive during restore fix(runsc): keep inventory responsive through slow restore Nov 5, 2026
@hky1999

hky1999 commented Nov 5, 2026 •

Copy link
Copy Markdown
Collaborator Author

Implemented the remaining inventory blockers in 2cfd3f6, on top of the upstream atomic-write backport 3a6a1cd. This supersedes the earlier TryLock-only acceptance failure reported in this PR.

The fix covers the full runsc list path: atomic metadata publication + read-only lock-free loading, bounded state-RPC refresh, transport cancellation and resource cleanup. Lifecycle mutation locks remain in place. A busy live Sentry is retained in inventory; a dead Sentry converges to stopped. Corrupt/unreadable metadata fails the command instead of silently omitting an entry.

Origin: the blocking state-file flock came from the old Google gVisor baseline. Latest Google upstream already includes related atomic publication and read-only loading work. This is not a Harbor/DinD-introduced bug, nor a claim that latest Google upstream still lacks all these fixes. The relevant changes are now adapted to this AKernel branch, with cancellable RPCs and bounded workers.

Local complete-bundle acceptance on Linux 5.15, isolated mount/PID namespaces, passed all 12 cases per platform: success, checkpoint-read EIO, Sentry exit, live nonresponding Sentry, and 8 simultaneous slow restores. Successful restores preserve the original nonce and advancing counter; failures/exits converge to stopped; the healthy sibling stays running.

Full runsc list Systrap KVM
Before (ed7c4148) during slow restore 9/9 exceeded 2 s in flock 9/9 exceeded 2 s in flock
After: 33 gated-restore calls 14.1–66.2 ms 13.1–33.0 ms
After: 3 nonresponding-Sentry calls 262.9–269.5 ms 262.4–267.0 ms
All transition/restore inventory calls 77 passed 79 passed

Final head a300cef807be7bee1340d1146cfaa10677001755 is green: CI run. Both architecture builds, all six ordinary regression suites and real Systrap acceptance passed. The three new amd64 race suites each passed all three repetitions. CI acceptance ran four simultaneous real slow restores per architecture; all failure/exit/paused-Sentry cases passed and cleanup inventories were empty.

Local six-suite regression, ten repetitions of each original new suite, serialization tests, and the complete bundle checks also passed. The production sources at the final head are unchanged from the locally accepted runtime revision 2cfd3f66a used for the Systrap/KVM eight-restore runs above.

For the audit trail: the initial CI's RPC race fixture failed because its artificial 8 MiB send generated approximately 32 MiB of failed-send diagnostics. A focused local reproduction confirmed the same failure (cancellation took 4.78 s, 33,555,108-byte log, no race report). The test-only follow-up starts cancellation after bytes reach the peer and abbreviates that synthetic diagnostic; the two-second cancellation assertion and race-detector output remain intact. The focused test passed three race repetitions, followed by the full final CI pass.

The one-second shared budget / 250 ms per query / 16 workers apply to inventory RPC refresh, not restore admission. They are not an absolute wall-clock guarantee under arbitrary host filesystem stalls. These results establish the targeted correctness/latency behavior, not production-scale capacity.

No production deployment changed and no release was published. Both isolated inventories ended empty, with no owned runtime processes, checkpoints, FUSE or temporary build mounts remaining. Test procedure: tools/runsc-list-restore-test.md.

EtiennePerot and others added 4 commits November 5, 2026 19:33
Opening the file for write truncates it, so this prevents creating
corrupt state file in case the process gets killed between opening
and writing, or when hitting disk-full errors (which it gets on the
write, not on the open).

(cherry picked from commit 9fa3d71)
Updating oom_score_adj for an exited, unreaped process could fail for a
nonroot caller: Linux makes the process's proc files root-owned after
exit [1], so opening the file failed with EACCES. The no-op-on-exit
handling added in 88f7369 covered ENOENT and ESRCH but missed this case.

Share the existing pidfd liveness check with the OOM-score helper and
ignore EACCES only after confirming that the process has exited. Preserve
the original error when the process is alive or its state is unknown.

[1]: https://github.com/torvalds/linux/blob/830b3c68c/fs/proc/base.c#L1848-L1869

Assisted-by: Codex
(cherry picked from commit 728f546)
Adds several tests to verify correctness of `runsc` operations that
deal with control commands which may hang. Ensure commands like
`runsc list`, `runsc state`, `runsc kill`, and `runsc delete` still
do their job regardless of that, instead of hanging forever.

Relevant for google#16139 which increases the risk of such conditions
happening, due to the introduction of a state where the sandbox is
in the middle of booting and has an open control channel socket
`listen`ing, but not yet `accept`ing calls.

(cherry picked from commit f76c600)
This refactors the way the state file is loaded such that read-only
usage does not require a lock. This solves the issue in google#16255, at
least in terms of avoiding spurious creation of `.lock` files by most
`runsc` invocations.

In addition, the Sentry PID matching is switched to be based on
PID + `pidfd` inode number match. This ensures that a reused PID
cannot accidentally be confused for a running sentry.

Both are relevant for google#16139: since booting becomes asynchronous,
Sentry process liveness must be robust (hence `pidfd` inode numbers),
and must depend less on RPC responsiveness (since there now exists a
window post `runsc create` where the control server may be genuinely
unresponsive but not for kill-worthy conditions). The use of read-only
state file access also enables `runsc state` and `runsc list` commands
to work smoothly during this time, and not risk hanging trying to
contact a booting sentry.

(cherry picked from commit bab1123)
@hky1999
hky1999 force-pushed the hky1999/runsc-state-query-20261025 branch from a300cef to 4710f90 Compare November 5, 2026 11:52
@hky1999 hky1999 changed the title fix(runsc): keep inventory responsive through slow restore fix(runsc): backport upstream lock-free and bounded inventory Nov 5, 2026
Cover concurrent atomic metadata publication, read-only loading, restore failure and Sentry exit, paused-Sentry fallback, and multiple FUSE-gated restores. Run privileged upstream command tests through the existing host sudo wrapper; keep runtime behavior entirely in the upstream cherry-picks.
@hky1999
hky1999 force-pushed the hky1999/runsc-state-query-20261025 branch from be127aa to e809115 Compare November 5, 2026 12:34
@hky1999

hky1999 commented Nov 5, 2026

Copy link
Copy Markdown
Collaborator Author

Reworked this PR to favor upstream provenance and future rebases. Current head: e809115b58cbb5163d31c70f3aef21e76197a142.

The history is now four cherry-pick -x commits (9fa3d71a6, prerequisite 728f546af, f76c600d6, bab1123c2), preserving authors, plus one test/CI commit. Only baseline compatibility resolutions remain. The prior custom try-lock, cancellable RPC APIs, inventory worker pool and one-second query budget are removed. Earlier results for the custom candidate should not be used to assess this head.

Validation: local complete bundle and eight focused suites pass; atomic metadata/identity suites pass ten repeats. Real FUSE-gated acceptance passes on Systrap and KVM, including success, read EIO, Sentry exit, paused live Sentry, eight simultaneous restores, state convergence, original-process checks and sibling visibility. The paused Sentry deliberately exercises upstream's approximately five-second fallback; the test deadline is seven seconds, not the old custom candidate's two seconds.

Isolated cluster acceptance: two rounds at each concurrency, each round deleting all sources before restoring. Every sandbox carries 64 MiB touched random memory and a 32 MiB writable file; original process identity, memory/file hashes, file offset and established internal stream state are verified.

Concurrent restores Restores and full proofs Checkpoint p50 / p95 Restore API p50 / p95 Full proof p50 / p95
300 600/600 3.80 / 4.38 s 0.55 / 0.74 s 3.97 / 8.71 s
1000 2000/2000 9.83 / 88.92 s 3.03 / 13.14 s 24.95 / 47.36 s

During restore through full proof, runsc list succeeded 120/120 and 666/666, with maxima 0.072 s and 0.781 s. Sandboxd List, health and available-runtime checks succeeded 118/118 and 675/675 each. No runtime reconciliation timeout/unavailable logs or OOM were observed.

Limitations and a separate finding: full proof includes the bounded proof queue; API return is not full page readiness. These calls use an isolated API through port-forward. The 1000 test places 50 sandboxes on each of 12 workers and 400 on one larger worker; page-cache reclaim pressure is recorded, so its timing is not a balanced capacity result. During the first checkpoint, that larger worker had five Sandboxd List API timeouts at the ten-second probe deadline. A 90-byte Sandboxd status-file fsync took 66.403 s in the same interval. Code inspection shows status publication holds the write lock used by List readers. In that interval runsc CLI list still passed 32 probes (max 0.141 s), and health/runtime availability passed. This supports a separate Sandboxd status-lock/durable-I/O issue; no blocked-goroutine stack was captured, and this gVisor PR does not change that path. We do not claim all lifecycle inventory APIs are timeout-free.

Bundle used for isolated/local acceptance: release-20261012.0-akernel.2-9-g9151bec1ae93, SHA256 d7578ed9e3ef8e46c43914f228436266529f3fafd741d116114c0ecbcd7d6532. The source tree is identical to final head except the CI workflow; final changes fix host-side privileged test execution and consolidate test commits. CI independently rebuilds the final head on both native architectures.

CI: 42028254289 — passed on both architectures. Complete bundles, focused regressions and real slow-restore acceptance pass; amd64 race suites pass all three repetitions. Acceptance evidence and race logs are attached to the workflow.

Cleanup completed: SDK environments/snapshots and all 13 runsc inventories empty; isolated namespace/control plane/Redis/workers/artifact server removed; all owned node directories emptied; local port-forward, monitors, temporary mounts and ephemeral private credentials removed. Production workload/service/configuration specs and data compare unchanged. No release or production rollout performed.

@rcyw
rcyw merged commit 49590af into akernel-dev:develop Nov 6, 2026
3 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants