Skip to content

fix(runsc): keep restore inventory responsive with opt-in admission limits - #81

Closed
hky1999 wants to merge 4 commits into
inclusionAI:mainfrom
hky1999:hky1999/runsc-state-query-20261025
Closed

hky1999 wants to merge 4 commits into
inclusionAI:mainfrom
hky1999:hky1999/runsc-state-query-20261025

Conversation

@hky1999

@hky1999 hky1999 commented Oct 24, 2026 •

Copy link
Copy Markdown
Collaborator

Concurrent restores can block gVisor state queries and put pressure on shared storage during filesystem preparation, metadata synchronization and memory loading. A temporary inventory failure can then reject accepted restores. This PR keeps runtime inventory responsive, waits for real inventory recovery within the request deadline, and provides an opt-in concurrency limit for runsc restore Start RPCs.

  • Pin the complete, qualified gVisor release release-20260817.0-akernel.6, including the state-query fix in fix(runsc): backport upstream lock-free and bounded inventory akernel-dev/gvisor#12. Runsc, the shim and all four helpers use the same verified architecture-specific archive. This PR does not rebuild the bundle.
  • Mount the disposable runsc host staging overlay with volatile. Guest writable filestores and checkpoint output synchronization retain their existing semantics.
  • Add plugin.runtime.runsc.restore_concurrency: omitted or 0 disables the concurrency limit. Only an explicitly configured positive value enables it; valid positive values are 1–1024. When enabled, a permit spans the complete restore Start RPC, including filesystem preparation, before sandbox ID or resource reservation. Cold starts, cleanup and other runtimes keep their existing paths.
  • An accepted runsc restore waits for a successful inventory poll within its existing request deadline, even when concurrency throttling is disabled. Advertisement stays withdrawn while inventory is unavailable; waiting does not fabricate an empty inventory or reopen health. Admission waits without a caller deadline remain bounded.
  • Preserve cancellation and deadlines while queued. Background page loading can outlive Start completion, so the optional limit bounds active restore RPCs rather than all background I/O.

No runtime polling timeout or Relay TCP connection timeout is increased. Coordinator changes are outside this PR.

Validation of current revision bf44cf21cb79877d349455561a869e2016ab8ec0:

  • A TOML-backed regression keeps 32 restore permits outstanding for both an omitted setting and explicit restore_concurrency = 0. Before this revision both configurations fail at request 17; afterward both pass.
  • Existing positive-limit tests verify queueing, permit reuse, original deadlines, cancellation and absence of sandbox ID reservation while queued. Invalid values remain rejected; an already cancelled context remains cancelled when throttling is disabled.
  • make check-fmt, make vet and make test pass; the full unit run reports 41 tested packages.
  • go test -race ./internal/server ./pkg/sandbox -run 'TestRestoreAdmission|TestWaitRuntimeHealth' -count=1 passes.
  • The public checkpoint/restore contract and configuration comments describe the disabled default and the independent health wait.

Historical qualification applies to the preceding candidate, not the new disabled default:

  • Source 35f1361 and HK companion da0ec5d used an implicit concurrency limit of 16. Their earlier PR checks, runtime qualification and cluster experiments do not establish performance or cluster correctness of the current unlimited default.
  • That candidate passed a five-environment canary, a 128-environment single-node checkpoint/restore cohort, and a 1000-environment full-cluster cohort. In the latter, all 100 task Dockerfiles were initialized ten times; all 1000 captures and verified restores succeeded. Restore submissions were concurrent but each Sandboxd admitted at most 16 active restore Start RPCs. The phase took 322.45 seconds including queueing, with P95 260.68 seconds. The pressure profile used 100m CPU/256 MiB requests with 2 CPU/8 GiB ceilings; it did not run LLM agents or verifier scoring.
  • Host I/O and TCP diagnostics still showed storage pressure; concurrency admission is overload control, not a storage performance fix. Optional Prometheus forwarding was unavailable in the final cohort, so no platform-metric coverage is claimed.
  • Historical cleanup verified empty owned environment/runtime inventories, deletion of all 1000 owned snapshots, preservation of seven pre-existing snapshots, removal of collectors and BuildKit pods, and restored DaemonSet strategies. The original Coordinator remained unchanged; candidate 04c9419 was not deployed and its admission issue is not claimed fixed here.

No production image, configuration, deployment or cluster workload was changed for the current revision. The previously deployed binary retains its old default behavior until a separately reviewed release is deployed. No new cluster qualification is claimed for unlimited admission.

Pin the complete amd64 and arm64 archives promoted unchanged from the
qualified candidate. Busy Loader state queries use the existing liveness
fallback instead of blocking node-wide inventory during kernel restore.

Record the temporary fork provenance and preserve the runtime availability
and checkpoint timeout contracts. Both published archives match the gated
candidate SHA-512 digests.

Signed-off-by: 月可 <hukeyang.hky@antgroup.com>
Copilot AI balanced review requested due to automatic review settings October 24, 2026 18:27

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has insufficient remaining quota.

An organization owner can choose the repository organization as the billing source for reviews requested by members with a paid Copilot license. Learn about Copilot code review billing.

Signed-off-by: 月可 <hukeyang.hky@antgroup.com>
(cherry picked from commit e92a7880766dc3fca44c983d05f3d3389ab1f57f)
@hky1999 hky1999 changed the title fix(runtime): pin gVisor with responsive restore inventory fix(runsc): keep restore inventory responsive and avoid teardown writeback Oct 25, 2026
Signed-off-by: 月可 <hukeyang.hky@antgroup.com>
@hky1999 hky1999 changed the title fix(runsc): keep restore inventory responsive and avoid teardown writeback fix(runsc): bound restore pressure and keep runtime inventory responsive Oct 25, 2026
Treat omitted and zero restore_concurrency as unlimited. Only positive
values enable the per-process Start RPC semaphore, while health waits,
request deadlines and cancellation keep their existing semantics.

Add TOML-backed regression coverage that holds more than 16 restores
for both disabled configurations, and update the checkpoint contract.

Signed-off-by: 月可 <hukeyang.hky@antgroup.com>
@hky1999 hky1999 changed the title fix(runsc): bound restore pressure and keep runtime inventory responsive fix(runsc): keep restore inventory responsive with opt-in admission limits Oct 26, 2026

hky1999 commented Nov 5, 2026

Copy link
Copy Markdown
Collaborator Author

Closing this earlier Draft because its Sandboxd-side behavior is included in the integrated, now-merged #98 (8760ca58).

The retained implementation includes inventory recovery within the caller's deadline, restore Start admission disabled by default/0 and enabled only by an explicit positive limit, and volatile staging-overlay handling with the older-kernel fallback. The older August runtime pin in this PR is superseded by the current complete release-20261012.0-akernel.3 bundle; it should not be independently reintroduced. The gVisor-side state-query proposal remains separately tracked in akernel-dev/gvisor#12. Further Harbor/runtime integration is tracked in inclusionAI/AKernel#148.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants