Repository navigation
fix(runsc): backport upstream lock-free and bounded inventory - #12
Conversation
|
Keeping this PR open as an independent runtime follow-up after consolidating the older Sandboxd/AKernel drafts. The currently tested HK1019 framework pins the complete release-20261012.0-akernel.3 runsc bundle. Its release tag resolves to 63de489. At that exact source, Loader.containerRuntimeState still takes l.mu.Lock(), ContainerRuntimeState does not propagate the transient busy error, and TestContainerRuntimeStateDoesNotWaitForLoader is absent: Line 2669 in 63de489 The current release therefore does not contain this PR's non-blocking state-query change. Merged inclusionAI/sandboxd#98 retains the Sandboxd-side inventory waiting/admission behavior, which is not a replacement for this Sentry/Loader fix. This PR remains separately tracked and needs alignment with the current runtime baseline, preserving its newer HasNonExitingTasks behavior. This deployment/source audit and PR consolidation do not change the running bundle or the main thread's active pressure test. |
0cb5583 to
ed7c414
Compare
|
Added the real slow-restore acceptance runner and recorded results in 01bf36d: instructions/results. The complete amd64 bundle from ed7c414 was exercised in isolated PID/mount namespaces, on both Systrap and KVM. Each platform covered success (same original workload nonce and advancing counter), checkpoint-read EIO, Sentry SIGKILL during restore, and subsequent inventory state convergence. All of those transition/convergence checks passed and final inventories were empty. Local config_test, boot_test and filestore_snapshot_test passed. The strict end-to-end inventory acceptance did NOT pass. Real state RPCs returned the expected transient busy error in 0.49–0.76 ms, but all 18 full I updated the PR description to separate the successful RPC fix from this remaining metadata-lock blocker. The test deliberately retains normal CLI locking and exits nonzero; it does not bypass the lock or drop a busy container from inventory. The new PR CI run is running; that workflow covers bundle/unit checks, not this manually executed strict FUSE acceptance. No production deployment was changed and no bundle was published. |
|
Implemented the remaining inventory blockers in 2cfd3f6, on top of the upstream atomic-write backport 3a6a1cd. This supersedes the earlier TryLock-only acceptance failure reported in this PR. The fix covers the full Origin: the blocking state-file flock came from the old Google gVisor baseline. Latest Google upstream already includes related atomic publication and read-only loading work. This is not a Harbor/DinD-introduced bug, nor a claim that latest Google upstream still lacks all these fixes. The relevant changes are now adapted to this AKernel branch, with cancellable RPCs and bounded workers. Local complete-bundle acceptance on Linux 5.15, isolated mount/PID namespaces, passed all 12 cases per platform: success, checkpoint-read EIO, Sentry exit, live nonresponding Sentry, and 8 simultaneous slow restores. Successful restores preserve the original nonce and advancing counter; failures/exits converge to stopped; the healthy sibling stays running.
Final head Local six-suite regression, ten repetitions of each original new suite, serialization tests, and the complete bundle checks also passed. The production sources at the final head are unchanged from the locally accepted runtime revision For the audit trail: the initial CI's RPC race fixture failed because its artificial 8 MiB send generated approximately 32 MiB of failed-send diagnostics. A focused local reproduction confirmed the same failure ( The one-second shared budget / 250 ms per query / 16 workers apply to inventory RPC refresh, not restore admission. They are not an absolute wall-clock guarantee under arbitrary host filesystem stalls. These results establish the targeted correctness/latency behavior, not production-scale capacity. No production deployment changed and no release was published. Both isolated inventories ended empty, with no owned runtime processes, checkpoints, FUSE or temporary build mounts remaining. Test procedure: tools/runsc-list-restore-test.md. |
Opening the file for write truncates it, so this prevents creating corrupt state file in case the process gets killed between opening and writing, or when hitting disk-full errors (which it gets on the write, not on the open). (cherry picked from commit 9fa3d71)
Updating oom_score_adj for an exited, unreaped process could fail for a nonroot caller: Linux makes the process's proc files root-owned after exit [1], so opening the file failed with EACCES. The no-op-on-exit handling added in 88f7369 covered ENOENT and ESRCH but missed this case. Share the existing pidfd liveness check with the OOM-score helper and ignore EACCES only after confirming that the process has exited. Preserve the original error when the process is alive or its state is unknown. [1]: https://github.com/torvalds/linux/blob/830b3c68c/fs/proc/base.c#L1848-L1869 Assisted-by: Codex (cherry picked from commit 728f546)
Adds several tests to verify correctness of `runsc` operations that deal with control commands which may hang. Ensure commands like `runsc list`, `runsc state`, `runsc kill`, and `runsc delete` still do their job regardless of that, instead of hanging forever. Relevant for google#16139 which increases the risk of such conditions happening, due to the introduction of a state where the sandbox is in the middle of booting and has an open control channel socket `listen`ing, but not yet `accept`ing calls. (cherry picked from commit f76c600)
This refactors the way the state file is loaded such that read-only usage does not require a lock. This solves the issue in google#16255, at least in terms of avoiding spurious creation of `.lock` files by most `runsc` invocations. In addition, the Sentry PID matching is switched to be based on PID + `pidfd` inode number match. This ensures that a reused PID cannot accidentally be confused for a running sentry. Both are relevant for google#16139: since booting becomes asynchronous, Sentry process liveness must be robust (hence `pidfd` inode numbers), and must depend less on RPC responsiveness (since there now exists a window post `runsc create` where the control server may be genuinely unresponsive but not for kill-worthy conditions). The use of read-only state file access also enables `runsc state` and `runsc list` commands to work smoothly during this time, and not risk hanging trying to contact a booting sentry. (cherry picked from commit bab1123)
a300cef to
4710f90
Compare
Cover concurrent atomic metadata publication, read-only loading, restore failure and Sentry exit, paused-Sentry fallback, and multiple FUSE-gated restores. Run privileged upstream command tests through the existing host sudo wrapper; keep runtime behavior entirely in the upstream cherry-picks.
be127aa to
e809115
Compare
|
Reworked this PR to favor upstream provenance and future rebases. Current head: The history is now four Validation: local complete bundle and eight focused suites pass; atomic metadata/identity suites pass ten repeats. Real FUSE-gated acceptance passes on Systrap and KVM, including success, read EIO, Sentry exit, paused live Sentry, eight simultaneous restores, state convergence, original-process checks and sibling visibility. The paused Sentry deliberately exercises upstream's approximately five-second fallback; the test deadline is seven seconds, not the old custom candidate's two seconds. Isolated cluster acceptance: two rounds at each concurrency, each round deleting all sources before restoring. Every sandbox carries 64 MiB touched random memory and a 32 MiB writable file; original process identity, memory/file hashes, file offset and established internal stream state are verified.
During restore through full proof, runsc list succeeded 120/120 and 666/666, with maxima 0.072 s and 0.781 s. Sandboxd List, health and available-runtime checks succeeded 118/118 and 675/675 each. No runtime reconciliation timeout/unavailable logs or OOM were observed. Limitations and a separate finding: full proof includes the bounded proof queue; API return is not full page readiness. These calls use an isolated API through port-forward. The 1000 test places 50 sandboxes on each of 12 workers and 400 on one larger worker; page-cache reclaim pressure is recorded, so its timing is not a balanced capacity result. During the first checkpoint, that larger worker had five Sandboxd List API timeouts at the ten-second probe deadline. A 90-byte Sandboxd status-file fsync took 66.403 s in the same interval. Code inspection shows status publication holds the write lock used by List readers. In that interval runsc CLI list still passed 32 probes (max 0.141 s), and health/runtime availability passed. This supports a separate Sandboxd status-lock/durable-I/O issue; no blocked-goroutine stack was captured, and this gVisor PR does not change that path. We do not claim all lifecycle inventory APIs are timeout-free. Bundle used for isolated/local acceptance: CI: 42028254289 — passed on both architectures. Complete bundles, focused regressions and real slow-restore acceptance pass; amd64 race suites pass all three repetitions. Acceptance evidence and race logs are attached to the workflow. Cleanup completed: SDK environments/snapshots and all 13 runsc inventories empty; isolated namespace/control plane/Redis/workers/artifact server removed; all owned node directories emptied; local port-forward, monitors, temporary mounts and ephemeral private credentials removed. Production workload/service/configuration specs and data compare unchanged. No release or production rollout performed. |
Slow restore can hold the lifecycle state-file flock or leave the Sentry control RPC temporarily unresponsive, blocking node-wide inventory on the older AKernel baseline. This PR now imports Google's fixes, rather than maintaining a second implementation.
Upstream changes
Base:
akernel-dev/gvisor:developat63de4896444eb140221c1284fa621394666224af.9fa3d71a600e1b89a6c64f27f206ee942fe41262: atomic state-file publication.728f546aff7d3005c2a8d373f68c5d7e963d629f: shared pidfd process-liveness helper, needed by the identity changes.f76c600d6ea58f6f43ffec982da69e69bd7b3f58: parallel list checks with a shared five-second wait; Created root containers use process liveness without querying a still-booting Sentry.bab1123c2e369d601bba5ae7f9bed24b96a77596: read-only lock-free loading, write guards and pidfd process identity.All retain the original authors and
cherry-pick -xprovenance. Conflict resolutions preserve upstream algorithms: mapsentryapi.RuntimeStateStoppedto the baselineboottype; retain older adjacent gofer fields and metric-registration code, applying read-only loading at the load sites that exist on this baseline. The upstream pidfs identity fallback remains usable on older kernels without pidfs.Removed the previous custom Loader TryLock, cancellable state-RPC APIs, 16-query worker pool, and one-second RPC budget. These are not implicitly necessary just because they existed in the earlier candidate. Prior comments and timing results for
a300cef8/2cfd3f66describe the superseded implementation, not this head.Contract and validation
Upstream list waits up to five seconds for live status, then loads saved metadata without the lifecycle flock. Sandboxd's inventory poll has a ten-second timeout. A direct state RPC can still block during restore; that diagnostic is distinct from bounded CLI inventory. Saved status does not imply workload readiness. Local filesystem stalls are not covered by the RPC wait budget. Upstream's log-and-skip behavior for unreadable records is retained.
Tests include atomic concurrent state-file readers and failed-publication preservation; upstream booting/unresponsive process, read-only guard and PID identity regressions; plus real FUSE-gated restore success, failure, Sentry exit, paused Sentry, and multiple simultaneous restores. The CLI acceptance deadline is explicitly seven seconds, rather than the earlier candidate's two seconds.
Measured validation
Head:
e809115b58cbb5163d31c70f3aef21e76197a142. Four upstream commits plus one test/CI commit; the earlier custom implementation has been removed. The amd64 complete bundle used below is stamped9151bec1ae93(SHA256d7578ed9e3ef8e46c43914f228436266529f3fafd741d116114c0ecbcd7d6532). Its entire source tree matches the final head except.github/workflows/akernel-pr.yml; subsequent changes only fixed privileged CI execution, then consolidated test commits. This is source-equivalent runtime validation, not a claim that the archive was rebuilt with the final Git stamp.Full proof is timed from restore submission and includes the bounded proof queue; it is not bare restore execution. Clients use an isolated API via kubectl port-forward. Dispatch spans are 0.13–0.14 s for 300-way restore and 0.48–0.64 s for 1000-way restore. The 300 cohort is balanced across workers; the 1000 cohort has 50 on each of 12 workers and 400 on one larger worker. The latter hits cgroup page-cache reclaim pressure (no OOM) and must not be presented as a balanced production capacity/performance result.
During restore through full proof: runsc CLI inventory 120/120 and 666/666 successful (max 0.072 / 0.781 s); Sandboxd List, health and available-runtime probes 118/118 and 675/675 successful per probe.
runscremained available; no periodic reconciliation timeout/unavailable log appeared in either complete cohort.Separate residual finding: across the entire 1000 lifecycle, Sandboxd List RPC timed out five times at its ten-second probe deadline, all during the first checkpoint on the 400-sandbox worker, not during restore. A 90-byte Sandboxd status write spent 66.403 s in fsync in the same interval. Sandboxd performs this write inside
statusStorage.UpdateSync's write lock while List needsStatus.Get's read lock. This evidence identifies a separate Sandboxd state-publication/I/O issue; this PR does not change it and does not claim all lifecycle inventory APIs are timeout-free. The runsc CLI remained responsive and health/runtime availability stayed successful even then.Final-head amd64/arm64 bundle, regression, gated-restore and amd64 race CI: run 42028254289 passed. Both architectures passed full-bundle builds, focused regressions and real gated-restore acceptance; amd64 atomic-metadata and process-identity race suites passed all three repetitions. The workflow retains acceptance and race-log artifacts.
Isolation and cleanup
No production deployment was changed; final comparisons of DaemonSet, Deployment, StatefulSet, ConfigMap and Service specs/data found no changes. The test used its own namespace, control plane, Redis, worker runtime directories and scoped SDK credentials. Both SDK catalogs and all 13 runsc inventories were empty at completion. All isolated Pods/namespace were removed, the 13 owned host directories emptied, port-forward/monitors stopped, and local temporary build mounts and ephemeral private credentials removed. Evidence and the complete candidate bundle are retained locally. No release is published by this PR; use the normal complete-bundle CI release process after review/merge.