Repository navigation
Description
A cold private OCI runner pull on AKernel develop can exceed the node's 30-second ADX RPC timeout even when the caller gives a substantially longer create budget. After the failed create, deletion and an empty directory do not immediately imply that node resources have stopped.
Versions:
- AKernel develop
98681e0bc74afeb0095d758b16f0ae3891156541, source-built linux/amd64 with runc enabled; sandboxd gitlink7f16edd2254c94d968bce3fcdea8a6df4e40116e. - Actual ADX/Execd release input:
20261012104710-9a7924658fbc(platform archive SHA-256b2a8f4a10b573167833fe58c8ad506b29e2bb544889b34da2e9920dbecde9831, Execd archive510bbcba558c689293b18d76a9c15b5f416ac51c037d0f0421088052f79bd443). The source-inspection gitlink is separately4b9d212bcec33a3918f6f9dc4ab9d395bfe08f42. - AKernel SDK installed from the above checkout (
0.1.6rc9), declaredadx-sandbox==0.1.0rc2; runc 1.5.1.
Reproduction on an isolated node (no QEMU/guest was started):
- Build with runc enabled, configure strict-TLS private-registry pull credentials, and use an uncached large OCI image (our Linux runner contains a shutdown VM archive; approximately 36.6 GB uncompressed).
- Construct
Sandbox(image=<immutable digest>, runtime="runc", extra_config={"enableKVM": True}, name=<unique>, schedule_timeout=600, inherit_entrypoint=False). AKernel derives nativecreate_timeout=660. - Image layers start downloading. API logs report
Sandbox create stream completed, duration30002 ms, statusThe operation was cancelled; adxlet recordsenvironment start timed out. - SDK transport retries retain the create request ID; subsequent responses say
environment create is not yet running. Constructor fails. - Named deletion returns successfully. Both
/api/instances(all pages) and/global-scheduler/scheduling_queueare empty; Redis contains the exact environment's Deleted receipt. At that same observation the node still has the correspondingrunc-sbox-<uuid>netns and the OCI import continues.sbox listandrunc listare empty, so those checks alone also miss the in-flight allocation.
The generated adxlet configuration has rpc_timeout_seconds: 30. This report does not claim a permanent orphan: late-arrival reconciliation may eventually finish cleanup. We retain the allocation journal and block a second launch until independent node checks confirm termination. Normal small-image runc commands, binary file roundtrip, real KVM ioctl, and fully verified deletion passed on this same node.
Expected follow-up:
- Propagate the remaining create deadline to runtime/image preparation, or document/configure the independent node timeout so a caller's long budget is usable for cold imports.
- Expose an observable create/cancel operation identity and terminal resource-cleanup state. A Deleted result should distinguish still-in-flight runtime allocation/import cleanup, or wait/fence it before reporting termination.
- Add a delayed-image-import regression covering create timeout, delete, late runtime arrival and independent netns/mount/process cleanup.
The evidence above is from an isolated local experiment. No shared service was restarted, no image was published and no cloud resources were modified. Guest authentication/UI acceptance is outside this report.
Activity
rcyw commentedon Oct 14, 2026
Follow-up after observing this same failed allocation to completion: OCI layer import finished after about 79 seconds; sandboxd then rejected StartSandbox with
context canceled. The netns eventually disappeared without restarting the service. A single empty runc state directory remained (runc list --format jsonreturned null but also warned that the container did not exist). We verified that directory was empty and removed only that task-owned directory withrmdir. Independent checks then confirmed the exact Deleted receipt plus empty sbox/runc, no QEMU/runc-shim processes, no netns, no private runtime mounts, and no runc state paths.This is eventual cleanup, not a permanently running orphan. The important gap is the interval between published Deleted and completion of the in-flight runtime work, plus the residual empty state directory. The validation now continues with task-local API/node RPC timeouts explicitly raised to 900 seconds after draining the isolated node; all pre-existing services are untouched.
rcyw commentedon Oct 14, 2026
Suggested repair scope and acceptance criteria (not a confirmed attribution of every failure to one component):
platform/adxlet/src/bin/adxlet.rs,controller/lifecycle.rs,runtime_control.rsandreconciliation.rs. Either honor the remaining create budget or explicitly expose and document the independent preparation timeout. Raising the task-local RPC timeout to 900 seconds is a workaround, not a lifecycle fix.builder/node.Dockerfile(and matching runtime/Execd input if required). Updating only the agent-dx gitlink does not change the binary exercised by this reproduction.Suggested regressions: a controlled slow cold import that exceeds 30 seconds but remains within the advertised create budget; cancellation/deletion during import; a late create completion after deletion; and retries using the same request identity. Assert eventual termination and no environment revival, held resources, netns, runtime mounts, processes or residual runc state directories. Check the advertised cleanup-complete state against those independent runtime observations, not just directory absence or delete ACK.
This AKernel issue can remain the integration tracker while component fixes are linked from ADX and, if investigation warrants it, sandboxd. No product fix has been validated by the current timeout workaround.
WenYuLuo commentedon Oct 14, 2026
The ADX fix has been committed and pushed to
refactor: ce80a189f352f1ee2ebb6a9f30ee218f2ae896d3, based on0a5dea4c0674c8930de78d787dc59bab1bc8d770. This is source delivery; it is not included in an AKernel binary release yet.The create entrypoint now fixes one absolute deadline and carries it through both local-first admission and central scheduling, including Coordinator → adxlet and sandboxd Start. Each hop uses the remaining budget. Cold image/backend preparation uses the total create budget;
initCallTimeoutSecondsbounds Execd initialization/readiness after backend preparation. Ordinary connection and control-call timeouts remain separate. This requires synchronized clocks and upgrading the API Server, Coordinator and adxlet together.Within that original deadline, a retry with the same request identity and arguments can drive an existing Pending assignment again. It retains the observed node/generation and uses the existing serialized adxlet controller, so a delivery failure or lost reply does not allocate another execution. Transport failure remains an unknown outcome. An existing terminal Failed result is returned as a non-retryable failure for create replay; creating a new attempt is separate from replaying that failed attempt. Existing restart/recovery policies retain their own behavior.
Validation: 335 non-ignored component tests passed. The real Redis/socket control-RPC suite passed all 31 tests, with network-mode and mTLS coverage. The new Coordinator → adxlet regression first fails delivery, then cancels the caller after backend Start has begun, and retries the same assignment. A controlled 3.2-second backend preparation succeeds despite a 3-second Coordinator RPC cap and a 1-second initialization budget, with exactly one Start. Separate sandboxd gRPC tests cover preparation beyond the node RPC cap and an expired deadline issuing no Start.
The default
make rust-check, documentation checks and diff checks pass. The full workspace all-features check is blocked locally by the unrelated AFS/FUSE build prerequisites.Cleanup still uses the existing late-runtime reconciliation path. It can continue after the caller's deadline or a logical Deleted result. This patch does not establish immediate physical cleanup; the reported large-image case and independent netns/mount/process/runc-directory checks still need deployment acceptance.
For AKernel integration, the actual ADX release archive and checksum used by its image build still need to be updated after packaging the fix. Updating the source gitlink alone is insufficient. This issue should remain open for that integration and acceptance work.
fix(platform): preserve create deadlines and retry pending assignments
rcyw commentedon Oct 15, 2026
Deployment-validation follow-up: the create-deadline and eventual-cleanup portions now have local runtime evidence, but I recommend keeping #96 OPEN for the remaining deletion/physical-completion contract.
Fixed inputs actually exercised
9f24718638131b09ad03a274b9ba52ff76396a38(PR fix(runtime): fence expired runc startup and retain failed deletions #99 squash), sandboxdd5886b42ceff89bf0bf9baf7f73f5c534f689dee(PR feat: configure shared ChunkDB capacity in node deployments #74 squash), ADX platform/Execdce80a189f352f1ee2ebb6a9f30ee218f2ae896d3.sha256:5d25768d05ca714803b95c685a915e0394a9b51998e88e401bac27f0c869fc2e. Twenty-three actual binary/rootfs hashes were rechecked against the previously audited image. Platform archive SHA-256:2639ff62149ce915a173d5b205309d5bcfa8e85c49495bb453be024e791a093d; Execd:c169abc604e19f69ffc9d4e9714ed4e4609d0510a92a533ebd0a1263c1af882c. This is actual archive integration, not just a source gitlink check.0.1.6rc9installed from that exact AKernel Git revision,sdk/python; actual backendadx-environment==0.1.0rc1. Native runc 1.5.1 was included and enabled. ADX RPC/deadline defaults were retained; no 900-second workaround.netns_cache_size=0. The cold runner digest wassha256:b584e2bc6173fcd065df33d15fba050473ab07289be284d061d323cd467e4d8b(approximately 36.6 GB uncompressed). No OCI entrypoint, QEMU or guest was started.What has been verified
OUTCOME_UNKNOWN / unknown,Timeout expired. This was not a client read timeout. The deadline-only trace also has sandboxdcontext deadline exceeded; with zero public DELETEs, all physical witnesses became empty at approximately 96.766 seconds and remained empty for over 30 seconds. No Running revival was observed. The original driver incorrectly required the literal worddeadlineand exited 1 after observation; that failure is retained, and the recorded response/timeline were separately checked against the pinned structured-error contract. The corrected expiry-then-delete run independently exercised the same server deadline.rmdir, or service restart. However, this ordering reproduces the remaining gap below.The previous campaign separately demonstrated a cold create succeeding after approximately 210.810 seconds with the unchanged ordinary 30-second RPC configuration. That success was not rerun in this follow-up.
Remaining reproducible gap: expire first, then delete once
Native request:
POST /api/environment/v1/environments, unique name/request ID,runtime=runc, immutable image,inheritEntrypoint=false,failover=false, create/schedule/init budgets 35/1/1 seconds. Once the server response and Failed state were observed while preparation resources remained, send oneDELETE /api/environment/v1/environments/{id}. Disable client retries/redirects; observe without any further mutating request.Exact environment:
default-macos-sdk-af43f22dae6247ce8a66c2b45272d5fe; generation 1; create request IDcreate-5efdfc25-a941-4750-9ca8-4fdb20a677df.Times below are relative to test start; POST was issued at approximately 0.168 seconds.
There are 58 samples with the exact generation-1 Deleted receipt and
resources_held=false, empty inventory/queue and empty sbox/runc lists, but netnsrunc-sbox-bebd970e-bc35-4375-b85f-2f3c2c03dabbstill present. Receipt result digest:9dfbd7968cec26755acb3dcd69f3263605f7ada83ae62c6b5041f3e4d02ce39b. Processes, runtime mounts, state/bundle/private-storage paths and runc cgroups were empty in those samples. Final independent verification confirmed that the netns also disappeared. Observation DELETE count 1; post-observation catalog-retirement DELETE count 0.Thus ACK precedes observed physical completion by approximately 66.297 seconds, and the first observed Deleted receipt precedes it by approximately 65.211 seconds. This is eventual cleanup, not a permanent orphan.
resources_held=falseis a scheduler-reservation fact and is not, by itself, a physical-cleanup completion guarantee. The internal Deleted receipt is also not a separate public cleanup-complete operation.Suspected boundary, not a proven unique root cause
At the exact packaged ADX source:
execute_start, lines 294–348 sets StartState::Failed before the RPC; an RPC error can leave that state after its task releases the lock.remove, lines 1152–1197 has the unknown-start guard for Uncertain, while Failed can return success when no published runtime identity/list entry is found.Controller::delete, lines 948–972 publishes Removed/Deleted after cleanup returns.instance_view, lines 956–980 then returns 404 and exposes no separate physical-cleanup status.This path is consistent with the observed ordering, but the current trace does not correlate every internal RPC/preparation/rollback boundary and does not prove it is the sole cause. It is also consistent with the existing logical-deletion limitation already acknowledged in the ce80a189 delivery comment, not a newly attributed regression in sandboxd #74.
Suggested remaining work and acceptance
cleanup pending/complete/failedstate with durable ownership/identity. Whichever contract is chosen, its completion signal must agree with independent netns/mount/process/state/bundle/private-storage/cgroup observations. An empty published-runtime list cannot by itself fence unpublished in-flight preparation.This follow-up is local native-runc deployment evidence, separate from remote CI and author component tests. Default Standalone E2E does not enable runc or establish this contract. Sampling was approximately once per second and does not exclude every sub-second transition. KVM/QEMU/macOS guest acceptance was not rerun here. Tool assertion/preparation failures and raw traces are retained locally, not uploaded; the table above is a sanitized excerpt. All four owned fixtures/networks were removed only after physical verification, with no task loop device, listener or allocation journal remaining; existing services, source VM and source disks were protected. No product code, image publication or cloud resource was changed.
Recommendation: keep #96 OPEN for this remaining contract/observability item. If it is split into a linked follow-up, explicitly agree on that scope before closing this tracker; final eventual cleanup or Delete ACK alone does not establish the currently requested full acceptance.
rcyw commentedon Oct 15, 2026
Controlled-import follow-up to the previous deployment evidence: the previously suspected boundary is now reproducible with correlated stages. For this path, client Start cancellation and Deleted publication happen while the native Start handler is still blocked in image preparation; rollback completes only after the layer body is released. This establishes a concrete mechanism for the remaining gap, not the unique cause of every possible failure. Recommendation: keep #96 OPEN for the completion/ownership contract.
@WenYuLuo, the following evidence and proposed contract may help scope the next component fix. The existing deadline and late-runc-start fixes remain effective; this is not attributed as a new regression in sandboxd #72/#74.
Inputs and reproduction
9f24718638131b09ad03a274b9ba52ff76396a38, sandboxdd5886b42ceff89bf0bf9baf7f73f5c534f689dee, ADX platform/Execdce80a189f352f1ee2ebb6a9f30ee218f2ae896d3. These are the actual integrated inputs from the previous comment, including the packaged ADX fix, not a gitlink-only candidate.sha256:5d25768d05ca714803b95c685a915e0394a9b51998e88e401bac27f0c869fc2e. Each node rechecked the actual sandboxd binary SHA-256:733780c1a72549e89b4e4206f3068e52375d22f658e575df69fca460ddbe7207. This phase reused the image; it did not rebuild it or rehash the entire payload again.adxlet.sandboxd_socketthrough a transparent, byte-for-byte UDS observation proxy. Use the native API with retries/redirects disabled, HTTP operation timeout 300 seconds, and create/schedule/init budgets 35/1/1 seconds,runtime=runc,inheritEntrypoint=false,failover=false. No product source or binary was modified.Send one
POST /api/environment/v1/environmentswith a unique name/request ID. After the real server deadline response and Failed state are observed with the layer still blocked and the preparation netns present, send exactly oneDELETE /api/environment/v1/environments/{id}. After observing the matching generation's Deleted receipt, public 404 and empty complete inventory/queue, keep the body blocked for another 15 seconds (A) or 30 seconds (B), then release it. Make no further public mutating requests. Observe physical cleanup for at least 30 stable seconds and verify the exact terminal identity independently.The create request/operation ID is associated with the environment/runtime/generation through the API record and the single observed native Start; DELETE has its own request ID on that same execution.
create-phase-8cf2a0b3-32bc-4483-b2ce-8f57c62ac648create-phase-8c1ba383-792f-4b99-9b4e-81f31a02d223default-macos-sdk-a424223c4de0498dbcdd03095e179be5default-macos-sdk-832f90a923ce4609abfb3cd87dc133fb-1/1-1/1e8a2dfcbddac3bf71cb42d08fde9d617b2e37fcb885c8f79c5c542af2059dafb1 / 01 / 01 / 11 / 1Observed order
All times below are seconds from public POST begin, unlike the previous comment's driver-start origin. Receipt writes use Redis MONITOR server timestamps; import/rollback use the native log timestamps; syscalls use the host clock. The native handler interval is bracketed by successful metrics polls, not inferred from the time a log watcher read the line.
OUTCOME_UNKNOWN / unknown / Timeout expiredThere are 43 A samples (35.459–50.609 seconds) and 84 B samples (35.563–65.605 seconds) with public 404, empty full inventory/queue, the exact Deleted receipt, empty ADX held resources and resources_held=false, while the layer body is still unsent, native Start completion count is zero, and the preparation netns remains. Sampling was approximately every 0.35–0.38 seconds; RPC parser errors and metrics probe errors during both runs were zero.
The netns identities were A
runc-sbox-f8aec300-4c3d-4b9e-82ca-a4e0decbd155and Brunc-sbox-03a045d3-c2bd-4d46-94f9-7ea0a99ed636. Saved syscall traces showumount2(..., MNT_DETACH) = 0followed byunlinkat(..., 0) = 0for those exact paths, after layer release. MNT_DETACH alone is not the whole release proof; it is combined with handler completion and the subsequent stable independent physical probes.B's RPC cancellation moved only 0.0005 seconds, and Deleted only 0.095 seconds, relative to A. Layer release moved 14.990 seconds and first physical all-empty moved 14.993 seconds. Cleanup follows the controlled import release, rather than a fixed delay after Deleted. The same native trace ID connects each Start receive and its later context-canceled failure. No runc workload/shim/QEMU process was launched, and no residual empty runc state directory was found.
Concrete source boundary
server.go:1392–1402starts filesystem/resource preparation and waits for both;fsmanager.go:106–107has no caller context.MountOCI:1023–1025uses an HTTP Post without that context, and the image worker usescontext.Backgroundat line 78. The path does not wait for a canceled Start's rollback before the client RPC ends.execute_start:310–317sets Failed before the RPC; a transport error leaves that state.remove:1152–1197guards Uncertain, but Failed can return Ok when no physical mapping or published runtime list entry is found. An empty published-runtime list does not include this unpublished preparation allocation. The observed native Delete count of zero is consistent with this path.Controller::delete:958–972publishes Removed/Deleted after cleanup returns Ok. It therefore precedes the in-flight native operation's completion in this experiment.runc Start cancellation gate:169–174correctly rejects the late result;server.go:1373–1388performs deferred rollback/resource release. TheStart result counter:36–42increments only after the handler, including its defers, returns. Its poll interval is a completion bound, not an exact timestamp within that interval.Together, the controlled intervention and fixed source establish this path: client RPC canceled → Failed → no published runtime interpreted as successful removal → Deleted → import finishes → canceled runtime launch rejected → deferred rollback/resources released → native handler completes. This combines a cancellation-propagation gap with a completion-confirmation gap; it is not evidence of a permanent orphan, nor proof that sandboxd alone explains every case.
Suggested fix and acceptance
cleanup pending/complete/failedstate, retain ownership and failure evidence, and ensure the complete signal agrees with independent physical observations.resources_held=falseremains a scheduler-reservation fact.Both diagnostic runs actually completed locally; this is separate from remote CI and author reports. The harness's 7 offline tests passed, but they do not establish product acceptance. Full large-image successful cold import, normal runc smoke, real KVM/QEMU/macOS guest acceptance, and restart/failure-recovery scenarios were not rerun in this controlled phase. Earlier passed layers retain their original version/scope.
Sanitized times/identities above are the evidence excerpt. Raw traces, configs, frozen source/hashes and a 54-file SHA-256 manifest remain local under
artifacts/experiments/2026-10-15-issue96-phase-correlation/(Aa-hold15, Bb-hold30); they are not uploaded or backed up off-host. The diagnostic harness and durable result are saved as uncommitted reviewable workspace changes, not yet a published product fix. The initial trace-ID parser returned null; a separate pre-teardown audit extracted only native line/time/TRACEID and confirmed the matching receive/failure IDs. Preparation/tool failures were retained separately, not counted as lifecycle success.After independent exact-generation terminal/inventory/physical rechecks, both owned nodes/bridges were removed and their tracers, ports and loop devices released, with zero additional public DELETEs. Existing services/source VM/disks were protected. No service restart/stop was used to obtain cleanup success; owned-node teardown occurred only after the stable physical proof. No product code, image publication or cloud resource was changed. The remaining completion/ownership contract is still unimplemented and unaccepted; eventual cleanup and Delete ACK alone are not grounds to close #96.
WenYuLuo commentedon Oct 15, 2026
Follow-up to the controlled-import diagnosis: the timeout reporting and OCI cancellation-propagation fixes are now submitted, with component regression and isolated native-runc acceptance completed. Keep #96 open for AKernel artifact integration and the remaining agreed acceptance scope.
master, head9ce1e4ab57689ac6ee71669af067349dd2058030: confirmed create deadlines return HTTP 504 /DEADLINE_EXCEEDED, with the active phase inmessage, for exampleenvironment initialization deadline exceeded during backend preparation/runtime start. The phase is retained on failure replay. The original absolute execution deadline is preserved; the bounded response-reporting allowance does not extend the execution budget. Confirmed terminal failures are not automatically retried; genuinely unconfirmed publication or lost replies retain their appropriate same-operation/unknown-outcome handling.main, headb56f5ed425fab2ef0d4d9ae890ee95d67d000086: propagate the Start context through filesystem/rootfs preparation, OCI rootfs and image-volume mounting, inherited process metadata lookup, the image-service facade and HTTP mount endpoint. The registry client now passesremote.WithContext(ctx)to the registry library; previously its ctx was only used for tracing. Layer workers and shared-resource waits also honor cancellation, while preparation results are collected before rollback releases ownership. Cleanup runs independently of the cancelled Start context. Live shared consumers remain usable; cancelling one HTTP/2 request no longer closes unrelated streams on that connection. This covers regular OCI preparation; S3/FUSE daemon startup and Nydus bootstrap retain their existing behavior.The reproduced mechanism is therefore addressed at both boundaries: a confirmed deadline identifies the failing phase, and cancellation interrupts blocked OCI reads instead of waiting for the registry body to be released. Existing logical deletion and eventual reconciliation semantics are retained.
Inputs actually exercised
AKernel
9f24718638131b09ad03a274b9ba52ff76396a38, real Redis/API Server/Coordinator/adxlet, native runc 1.5.1 enabled, and the rebuilt sandboxd #75 binary. ADX acceptance used rebuilt binaries containing the !51 timeout fix from its pre-retarget source (4e6abe2plusa452f65); affected production source hashes were verified to match the patch retargeted tomaster. These are task-built binaries, not a newly published ADX archive integration.Final isolated image ID:
sha256:283ac86ba06b6f7c133744edd8077296e1ea76e97b74d46c13513e19063c29b0. The running sandboxd binary was verified against the built SHA-256:84f67f3c56d9318d2422e37f940c8546c8c6afcb0491f2945703515692d0cad8.The controlled registry serves valid, distinct OCI digests. Native runc is enabled and netns caching disabled. Create/schedule/init budgets are 35/1/1 seconds for expiry cases; the successful control uses a 10-second init budget. The registry flushes response headers and withholds the layer body. There is no elevated 900-second RPC workaround.
errorCode=504All three expiry cases report
DEADLINE_EXCEEDED / never / terminalwith the backend preparation/runtime start message. Their registry readers disconnect on cancellation; the layer bodies are never released. Independent sbox/runc lists, netns, runtime mount/state paths and process observations become empty and remain stable for more than 30 seconds. No late Running is observed. There is exactly one public DELETE per case, no additional cleanup DELETE, manualrmdir, or service restart to obtain cleanup success. The task container is removed after verification and the pre-existing node remains running. The expiry case also checks failed-create message replay.The early DELETE still waits behind the serialized create until its deadline; that explains the 32.845 seconds above. These fixes do not change lifecycle queue ordering.
Regression gates
make rust-check, documentation and diff checks pass.make check-fmt,make vet,make testpass (41 packages pass, zero fail; eight packages have no tests). Cancellation/shared-consumer regressions in all five affected packages pass five repetitions undergo test -race, including blocked manifest/config/layer reads, Start rollback, attached OCI volumes, cancelled shared-rootfs creators/waiters, shared-layer contention and actual HTTP/2 stream isolation. The blocked-download and HTTP/2 tests both demonstrated the failures before their fixes.make release-binarysucceeds in the Ubuntu 20.04 build image with seccomp/apparmor tags; the final binary is the one used for the four native-runc cases above.This evidence covers the controlled OCI cancellation path on the dedicated development host, separately from CI and published-release acceptance. AKernel's actual release/archive/checksum inputs still need promotion and integration after review. Full large-image successful import, KVM/QEMU/macOS guest acceptance and restart/rollback-failure recovery were not rerun in this campaign. No physical-cleanup acknowledgement contract is inferred from a DELETE ACK.
Sanitized results are preserved locally in
output/issue96-cancel-native-summary.json; the development-host log is/var/log/sandboxd-issue96-cancellation/native-final-3.log, with the fixture and raw evidence under/root/sandboxd-worktrees/start-preparation-cancellation-issue96/output/issue96-cancel-native-final/. These paths are retained evidence, not public artifact downloads.fix(server): preserve backend error categories and timeout stages
fix(platform): preserve backend timeout stages and error categories
WenYuLuo commentedon Oct 15, 2026
Follow-up: the early backend timeout chain and non-timeout error classification are now fixed in the existing PRs, with component checks and real public-API standalone regression completed.
master, head17d4833160e507e7352b2ac430c368a142042f6a(production fix075af5c, followed by E2E-driver checks).main, head02386fea2050c4a77cfaa9df5edfe72fd35fc987.Timeout propagation
sandboxd now preserves typed network/image-pull and runtime-start deadlines, including an internal timer that expires while the overall Start/create budget still has time remaining. The error message retains the concrete stage and original cause. runsc RPC timer errors and the runc startup-wait timer retain
context.DeadlineExceeded; filesystem preparation labels rootfs/image-volume pulls and inherited process-metadata lookup separately.ADX preserves that application timeout through adxlet, Coordinator and API Server. A confirmed, durably published timeout returns
DEADLINE_EXCEEDED / never / terminal, with HTTP 504 for JSON orerrorCode=504in the SSE final event. Both same-request replay and authoritative failed-record replay retain the exact message. The absolute execution deadline is unchanged; uncertain transport/lost-reply or unconfirmed-publication results retain their existing retry/outcome contract.Actual message excerpts from the native run:
sandboxd: filesystem preparation: image pull (rootfs): context deadline exceeded: ... net/http: TLS handshake timeout.sandboxd: runtime start (runc): waiting for runc container start after 10s: context deadline exceeded.The internal
sandboxd-start-failuretrailer identifies an application deadline reply; it is not a physical-cleanup completion acknowledgement. Cleanup continues through rollback/reconciliation independently of the cancelled request context.Non-timeout errors
Other backend gRPC errors retain their category and original message instead of being flattened to
Unavailableor inferred to be a timeout. Explicit cancellation takes precedence over network-timeout normalization, and actual connection failures remain transport failures. Classification uses typed errors/status codes.The real registry controls verify both JSON and SSE:
UNAUTHENTICATEDnever / not_startedPERMISSION_DENIEDnever / not_startedNOT_FOUNDnever / terminalRESOURCE_EXHAUSTEDnever / terminalEach response retains the original registry rejection message. Backend registry 429 is distinguished from node admission/capacity rejection: Coordinator returns the backend failure directly; node admission retains
after_backoff / not_startedand its existing retry behavior.Regression results
The final run uses AKernel
9f24718638131b09ad03a274b9ba52ff76396a38as an isolated standalone fixture with real Redis, API Server, Coordinator, adxlet, rebuilt sandboxd and native runc. These are public ADX API calls in that fixture, with SDK regression checked separately.All four internal-timeout cases return the precise stage and
DEADLINE_EXCEEDED / never / terminalbefore the total deadline. SSE transport is HTTP 200 with finalerrorCode=504. Exact-message replay passes for the original Request ID and a new Request ID resolving the same failed Environment. The TLS timings above belong to this controlled blackhole/retry fixture, not a universal image-pull timeout setting.All eight non-timeout registry controls pass. Healthy create/delete succeeds before and after fault injection. Final independent sbox/runc inventory, netns/state-directory, mount and runtime-process audits are empty. The disposable container is removed and the pre-existing node remains running; running binaries are checked against the fixture artifacts. The tested sandboxd SHA-256 is
eb71d87a96e481b23761d2865b2624d43ca713a9e15105067248a186b5d104cb.Additional gates:
-O; the corrected driver passes the complete native run above.02386fe, including all seven runtime E2E variants. The early internal runtime-timeout fixture above specifically exercises native runc; runsc's internal timer also has a component regression.17d4833(bot result194038829, 2026-10-15 22:22 +08:00).Final native evidence is retained on the development host under
/root/adx-issue96-timeout-20261015/out/internal-timeout-native-codecheck/; log:/var/log/adx-internal-timeout-native-codecheck-20261015.log. These are retained local evidence paths, not public downloads.Keep #96 open for reviewed artifact promotion/AKernel release integration and the agreed remaining acceptance scope. This follow-up records source-built standalone acceptance; it does not establish published-release or cluster deployment acceptance.
fix(platform): report create timeout phases
!51 merge fix/create-timeout-phase-issue96-master into master
build(sandboxd): pin Start cancellation and timeout fixes
rcyw commentedon Oct 16, 2026
Post-merge deployment acceptance completed on 2026-10-16 (Asia/Shanghai). The original single-node cold-import/deadline/cancellation report is covered on the fixed AKernel PR #100 squash below. Closing #96 as completed under the eventual physical cleanup contract, with the limits and original harness failures recorded here.
This supersedes my earlier suggestion that logical Deleted must wait for physical release or require a new public cleanup API. Logical deletion/scheduler release and physical reclamation remain separate; background rollback/reconciliation must eventually release the exact allocation without another public DELETE or a service restart. Delete ACK, logical Deleted, empty inventory, and completed physical cleanup were observed separately. Their timing need not be strongly consistent.
Actual integrated inputs
b5db829f9da17c970d4498d460f235fbdb8cea1c; source-installed SDK from its exactsdk/python, version string0.1.6rc9, actual backendadx-environment==0.1.0rc1(noadx-sandbox).02386fea2050c4a77cfaa9df5edfe72fd35fc987. It is obtainable from the canonical source and has the same tracked tree (716a846fcc3025fb739c7c6add3170bc97b99449) as merged sandboxd build: package the complete gVisor akernel.3 runtime bundle #75 squashff3c4d82b31b0ebd3ed99fa436816c3cbc28e292.af7d82f04a52d4604ea7ca4c550f09c6114dee4d. Actual platform archive SHA-25684f2cc11dca67aed642f4908c0e74ba1978d272491e239adad2ad03ba302da77; Execd archive62edc1079f5fcb500d25f397bf6c43bc7ff74b885ec8ec386069df138c2c8db1. Archive manifests, delivered node binaries, and Execd inside the runtime rootfs were checked, not just the source gitlink.sha256:dd88bef66249d0a9bb8f752e30a6a16a87aa0dbb9eab044e377bd5b89ca32a7c; native runc 1.5.1 included and enabled, with the shim/helpers present. Kata and Firecracker were disabled. sandboxd was compiled from the fixed checkout; the build excludes.git, so an embedded sandboxd VCS revision is not claimed.sha256:b584e2bc6173fcd065df33d15fba050473ab07289be284d061d323cd467e4d8b. Each cancellation fixture used its own initially empty OCI cache, node, data directory, network and exact ownership checks. ADX RPC defaults were retained; no 900-second workaround.This round's actual runtime evidence
runtime=runc, no inherited entrypoint, no transport retries, and a layer body that was never released. All returned the original request ID withDEADLINE_EXCEEDED / never / terminal; the registry reader disconnected at about 35 seconds. Exactly one native Start was observed per case, with no late runtime and at least 30 stable seconds after cleanup.Times in the following table are seconds from public POST begin. The physical column is a successful probe's sampling interval, not an exact release timestamp.
errorCode=504/ 35.003The early DELETE was sent at 0.499 seconds and acknowledged at 35.062 seconds: it still queues behind the create deadline, so immediate cancellation is not claimed. Native handler-completion poll bounds for the four cases were respectively
(34.9976,35.0496],(34.9881,35.0399],(35.0191,35.0711],(35.0119,35.0638]. Unlike the prior reproduction, handler completion/rollback no longer waited for manual layer-body release. For the two zero-DELETE cases, Failed catalog plus automatic physical cleanup was frozen first; explicit catalog retirement happened afterwards and is not used to prove automatic cleanup. The one-DELETE cases needed no subsequent public mutation.A separate real 15-layer empty-cache import cancellation sent exactly one POST and one DELETE. The create returned HTTP 504 /
DEADLINE_EXCEEDED / never / terminalafter 35.0026 seconds. DELETE was sent 5.714 seconds after POST begin (6.032 on the driver clock), and returned 200 at 35.046 seconds after POST begin. Actual native labels bound requestcreate-73e75fda-cfcc-4d5a-a97d-1a0dea1c6a50to environmentdefault-macos-sdk-4be916b5bdc6497fab05b6e9f16fb646, runtime environment ID +-1, generation 1; 32 assignment samples and Start counter 0→1 agreed. All 15 layers were cache misses and cancellation was logged. The exact Deleted receipt, public 404, complete inventory/queue, held resources and all physical items were empty for 30.297 seconds / 28 samples, without another DELETE or restart.This runtime harness originally failed after making those observations: it incorrectly required an already published
control.runtimeduring canceled preparation. The original failed run, summary, journal, logs and script are preserved. A separate read-only reassessment passed using the actual native Start labels/assignment/counter; it made zero new public mutations. Three validator regressions passed, accepting the complete unpublished-preparation identity and rejecting wrong environment/generation or missing runtime labels. Netns ownership is scoped by an initially empty exclusive node, its sole Start/assignment and sole preparation netns; a direct netns-to-label field is not claimed.The maintained opt-in
fault-runc-delete-retrytest actually ran: one passed, zero skipped. One public DELETE was followed by automatic reconciliation in about 0.406 seconds. Metadata, private-file content, runtime state, mounts, netns/cgroup, assignment and ADX held resources were retained at the injected failure and before the successful native retry. Init naturally exited after TERM; continuous process survival is not asserted. Catalog cleanup was recorded separately.Ordinary native runc command/stdout/stderr/nonzero exit and binary-file transfer passed; a real KVM ioctl returned
KVM_API=12 VM_CREATED=1. Invalid snapshot input was rejected before QEMU, followed by independent physical cleanup. Service TLS trusted/untrusted/wrong-host checks passed across control/resources, command, files, real command-watch handshake, PTY and CLI inventory/deletion.Two fresh, serial-use private-baseline macOS guest lifecycles passed trusted SSH and host-key checks, Darwin / macOS 15.8 / 24H23 / admin GUI, Web/RFB and six complete 1920×1080 PNGs, command streams/nonzero status, binary transfer, and actual mixed-case desktop positive/negative tasks. Both live QEMUs were correlated to the exact environment/generation/runc cgroup and held KVM VM/vCPU FDs. Both shut down normally and passed independent deletion,
forced=false. The 28 adapter runtime/tool/dependency hashes stayed unchanged through both runs.Controlled loss of one real successful create result retried the same operation/name, exposed one visible instance and incremented native Start only once. This is a controlled discarded return value, not a real TCP-failure test. Offline adapter regression: 82 passed, one skipped for missing host zsh.
Independent physical checks required empty sandboxd/runc inventory, no runtime/QEMU/shim processes, netns, private mounts, runtime state, bundle directories, private storage or runc cgroup leaves. Exact-generation terminal receipts and ADX held resources were checked separately from those physical observations.
Evidence retention, failures and final resources
Sanitized results are included above. Complete raw evidence and diagnostic sources remain local only, not publicly uploaded or backed up under
artifacts/experiments/2026-10-16-issue96-post-pr100/. The summary isvalidation-summary.json, SHA-256b474ba737a78f7f16a7f00f602608f90499b734fbeabfcaf915fcc7f6d69e1ae; its 24 evidence-file hashes were rechecked before posting. Main evidence entries:controlled-four-case-results.jsonand per-case frozen fixture sources incontrolled-case-sources/(17 tool regressions passed).20261016T131641068305-issue96-real-cancel-cold-single-delete-a8d89798df51; separate read-only reassessment20261016T132251766398-issue96-real-cancel-evidence-reassessment-9fd27e8d2eda.20261016T130257724036-issue96-post-pr100-authenticated-01-4fe810107b38and20261016T131436427284-issue96-post-pr100-authenticated-02-5f882a0400b1.20261016T132702011544-issue96-post-pr100-protection-final-0e42f286db1e; evidence/PNG/provenance summary audit20261016T132710841384-issue96-post-pr100-acceptance-summary-c950499aa3b7.A real-cancellation retirement wrapper also exited 1 on Docker “not found” diagnostic-text matching after its strict helper successfully removed the exact owned node/network. That failed run remains unchanged; a separate read-only final audit confirmed both IDs absent, ports refused and owned loops zero. Initial download-routing and overstrict empty-cache preflight failures are retained separately and are not counted as product passes. No failed run has been relabeled as successful.
All 7 task nodes and 7 networks were removed after physical proof; all 14 ports are closed, owned loop residue is zero, no host QEMU/runc-shim remains, and the private-baseline lock is released. The seven pre-existing containers including the source VM retained their identity/image/state/start/finish data. Original disk file-set/inode/device/size/mtime remained unchanged; this is a metadata check, not a new byte hash of the original 128-GiB disk. Private local inputs/images/evidence remain retained. No shared service was restarted/stopped, no source VM/disk was modified, no image was published and no cloud resource was changed.
Closure scope / remaining independent work
The original single-node cold-import budget, real deadline expiry, blocked OCI cancellation and eventual physical reclamation are accepted for this fixed local build. This does not establish synchronous physical release at Deleted or a new public cleanup-complete guarantee.
Not run in this round: daemon/control-plane restart, real unmount/rollback failure recovery, distributed clock skew, multi-node/multi-instance or shared-consumer stress, real TCP create-result loss, pause/resume, Kata/Firecracker/GPU, Nydus/S3/FUSE paths, cloud deployment, or acceptance of newly published release images/wheels. The author's early internal-timeout and non-timeout registry-error matrix remains separately attributed author evidence; it was not independently rerun as part of this campaign. Default CI/Standalone E2E is not used as a substitute for any native/opt-in/guest acceptance above. SDK security-default tracking in #97 and the native registry's existing
skip_verifybehavior remain separate from this closure; the adapter explicitly enables service verification.These broader scenarios can be tracked independently without extending this original report indefinitely. The historical observations and earlier open recommendations remain preserved; this comment records the accepted design and post-merge result used to close it.
fix(runtime): align managed profiles with current develop and hosts r…