Skip to content

ADX cold image create hits 30s node timeout; Deleted receipt precedes in-flight runtime cleanup #96

Closed

Description

@rcyw

A cold private OCI runner pull on AKernel develop can exceed the node's 30-second ADX RPC timeout even when the caller gives a substantially longer create budget. After the failed create, deletion and an empty directory do not immediately imply that node resources have stopped.

Versions:

  • AKernel develop 98681e0bc74afeb0095d758b16f0ae3891156541, source-built linux/amd64 with runc enabled; sandboxd gitlink 7f16edd2254c94d968bce3fcdea8a6df4e40116e.
  • Actual ADX/Execd release input: 20261012104710-9a7924658fbc (platform archive SHA-256 b2a8f4a10b573167833fe58c8ad506b29e2bb544889b34da2e9920dbecde9831, Execd archive 510bbcba558c689293b18d76a9c15b5f416ac51c037d0f0421088052f79bd443). The source-inspection gitlink is separately 4b9d212bcec33a3918f6f9dc4ab9d395bfe08f42.
  • AKernel SDK installed from the above checkout (0.1.6rc9), declared adx-sandbox==0.1.0rc2; runc 1.5.1.

Reproduction on an isolated node (no QEMU/guest was started):

  1. Build with runc enabled, configure strict-TLS private-registry pull credentials, and use an uncached large OCI image (our Linux runner contains a shutdown VM archive; approximately 36.6 GB uncompressed).
  2. Construct Sandbox(image=<immutable digest>, runtime="runc", extra_config={"enableKVM": True}, name=<unique>, schedule_timeout=600, inherit_entrypoint=False). AKernel derives native create_timeout=660.
  3. Image layers start downloading. API logs report Sandbox create stream completed, duration 30002 ms, status The operation was cancelled; adxlet records environment start timed out.
  4. SDK transport retries retain the create request ID; subsequent responses say environment create is not yet running. Constructor fails.
  5. Named deletion returns successfully. Both /api/instances (all pages) and /global-scheduler/scheduling_queue are empty; Redis contains the exact environment's Deleted receipt. At that same observation the node still has the corresponding runc-sbox-<uuid> netns and the OCI import continues. sbox list and runc list are empty, so those checks alone also miss the in-flight allocation.

The generated adxlet configuration has rpc_timeout_seconds: 30. This report does not claim a permanent orphan: late-arrival reconciliation may eventually finish cleanup. We retain the allocation journal and block a second launch until independent node checks confirm termination. Normal small-image runc commands, binary file roundtrip, real KVM ioctl, and fully verified deletion passed on this same node.

Expected follow-up:

  • Propagate the remaining create deadline to runtime/image preparation, or document/configure the independent node timeout so a caller's long budget is usable for cold imports.
  • Expose an observable create/cancel operation identity and terminal resource-cleanup state. A Deleted result should distinguish still-in-flight runtime allocation/import cleanup, or wait/fence it before reporting termination.
  • Add a delayed-image-import regression covering create timeout, delete, late runtime arrival and independent netns/mount/process cleanup.

The evidence above is from an isolated local experiment. No shared service was restarted, no image was published and no cloud resources were modified. Guest authentication/UI acceptance is outside this report.

Activity

  1. rcyw commented on Oct 14, 2026

    @rcyw
    CollaboratorAuthor

    Follow-up after observing this same failed allocation to completion: OCI layer import finished after about 79 seconds; sandboxd then rejected StartSandbox with context canceled. The netns eventually disappeared without restarting the service. A single empty runc state directory remained (runc list --format json returned null but also warned that the container did not exist). We verified that directory was empty and removed only that task-owned directory with rmdir. Independent checks then confirmed the exact Deleted receipt plus empty sbox/runc, no QEMU/runc-shim processes, no netns, no private runtime mounts, and no runc state paths.

    This is eventual cleanup, not a permanently running orphan. The important gap is the interval between published Deleted and completion of the in-flight runtime work, plus the residual empty state directory. The validation now continues with task-local API/node RPC timeouts explicitly raised to 900 seconds after draining the isolated node; all pre-existing services are untouched.

  2. rcyw commented on Oct 14, 2026

    @rcyw
    CollaboratorAuthor

    Suggested repair scope and acceptance criteria (not a confirmed attribution of every failure to one component):

    • Investigate deadline propagation across ADX apiserver, adxlet and runtime/image preparation. Relevant ADX areas include platform/adxlet/src/bin/adxlet.rs, controller/lifecycle.rs, runtime_control.rs and reconciliation.rs. Either honor the remaining create budget or explicitly expose and document the independent preparation timeout. Raising the task-local RPC timeout to 900 seconds is a workaround, not a lifecycle fix.
    • Investigate the create/cancel/delete race at the ADX–sandboxd boundary. Track in-flight creation until it finishes or is fenced; a late runtime result must be reclaimed. If logical deletion intentionally precedes physical cleanup, expose a distinct observable cleanup-complete state rather than making a Deleted receipt sufficient evidence of release. Verify sandboxd cancellation/rollback guarantees as part of this investigation; current evidence does not establish that sandboxd alone is the root cause.
    • AKernel should own integration and configuration regression coverage. After an upstream ADX fix, update the actual pinned release archive/checksum in builder/node.Dockerfile (and matching runtime/Execd input if required). Updating only the agent-dx gitlink does not change the binary exercised by this reproduction.

    Suggested regressions: a controlled slow cold import that exceeds 30 seconds but remains within the advertised create budget; cancellation/deletion during import; a late create completion after deletion; and retries using the same request identity. Assert eventual termination and no environment revival, held resources, netns, runtime mounts, processes or residual runc state directories. Check the advertised cleanup-complete state against those independent runtime observations, not just directory absence or delete ACK.

    This AKernel issue can remain the integration tracker while component fixes are linked from ADX and, if investigation warrants it, sandboxd. No product fix has been validated by the current timeout workaround.

  3. WenYuLuo commented on Oct 14, 2026

    @WenYuLuo
    Collaborator

    The ADX fix has been committed and pushed to refactor: ce80a189f352f1ee2ebb6a9f30ee218f2ae896d3, based on 0a5dea4c0674c8930de78d787dc59bab1bc8d770. This is source delivery; it is not included in an AKernel binary release yet.

    The create entrypoint now fixes one absolute deadline and carries it through both local-first admission and central scheduling, including Coordinator → adxlet and sandboxd Start. Each hop uses the remaining budget. Cold image/backend preparation uses the total create budget; initCallTimeoutSeconds bounds Execd initialization/readiness after backend preparation. Ordinary connection and control-call timeouts remain separate. This requires synchronized clocks and upgrading the API Server, Coordinator and adxlet together.

    Within that original deadline, a retry with the same request identity and arguments can drive an existing Pending assignment again. It retains the observed node/generation and uses the existing serialized adxlet controller, so a delivery failure or lost reply does not allocate another execution. Transport failure remains an unknown outcome. An existing terminal Failed result is returned as a non-retryable failure for create replay; creating a new attempt is separate from replaying that failed attempt. Existing restart/recovery policies retain their own behavior.

    Validation: 335 non-ignored component tests passed. The real Redis/socket control-RPC suite passed all 31 tests, with network-mode and mTLS coverage. The new Coordinator → adxlet regression first fails delivery, then cancels the caller after backend Start has begun, and retries the same assignment. A controlled 3.2-second backend preparation succeeds despite a 3-second Coordinator RPC cap and a 1-second initialization budget, with exactly one Start. Separate sandboxd gRPC tests cover preparation beyond the node RPC cap and an expired deadline issuing no Start.

    The default make rust-check, documentation checks and diff checks pass. The full workspace all-features check is blocked locally by the unrelated AFS/FUSE build prerequisites.

    Cleanup still uses the existing late-runtime reconciliation path. It can continue after the caller's deadline or a logical Deleted result. This patch does not establish immediate physical cleanup; the reported large-image case and independent netns/mount/process/runc-directory checks still need deployment acceptance.

    For AKernel integration, the actual ADX release archive and checksum used by its image build still need to be updated after packaging the fix. Updating the source gitlink alone is insufficient. This issue should remain open for that integration and acceptance work.

  4. rcyw commented on Oct 15, 2026

    @rcyw
    CollaboratorAuthor

    Deployment-validation follow-up: the create-deadline and eventual-cleanup portions now have local runtime evidence, but I recommend keeping #96 OPEN for the remaining deletion/physical-completion contract.

    Fixed inputs actually exercised

    • AKernel 9f24718638131b09ad03a274b9ba52ff76396a38 (PR fix(runtime): fence expired runc startup and retain failed deletions #99 squash), sandboxd d5886b42ceff89bf0bf9baf7f73f5c534f689dee (PR feat: configure shared ChunkDB capacity in node deployments #74 squash), ADX platform/Execd ce80a189f352f1ee2ebb6a9f30ee218f2ae896d3.
    • Local runc-enabled node image ID: sha256:5d25768d05ca714803b95c685a915e0394a9b51998e88e401bac27f0c869fc2e. Twenty-three actual binary/rootfs hashes were rechecked against the previously audited image. Platform archive SHA-256: 2639ff62149ce915a173d5b205309d5bcfa8e85c49495bb453be024e791a093d; Execd: c169abc604e19f69ffc9d4e9714ed4e4609d0510a92a533ebd0a1263c1af882c. This is actual archive integration, not just a source gitlink check.
    • SDK 0.1.6rc9 installed from that exact AKernel Git revision, sdk/python; actual backend adx-environment==0.1.0rc1. Native runc 1.5.1 was included and enabled. ADX RPC/deadline defaults were retained; no 900-second workaround.
    • Each scenario used a new, exclusively owned node with an empty OCI cache and netns_cache_size=0. The cold runner digest was sha256:b584e2bc6173fcd065df33d15fba050473ab07289be284d061d323cd467e4d8b (approximately 36.6 GB uncompressed). No OCI entrypoint, QEMU or guest was started.

    What has been verified

    1. A real 35-second server create budget expires during preparation. The response arrived after approximately 35.001 seconds with matching request ID and structured OUTCOME_UNKNOWN / unknown, Timeout expired. This was not a client read timeout. The deadline-only trace also has sandboxd context deadline exceeded; with zero public DELETEs, all physical witnesses became empty at approximately 96.766 seconds and remained empty for over 30 seconds. No Running revival was observed. The original driver incorrectly required the literal word deadline and exited 1 after observation; that failure is retained, and the recorded response/timeline were separately checked against the pinned structured-error contract. The corrected expiry-then-delete run independently exercised the same server deadline.
    2. During an early cold import, exactly one public DELETE was sent at approximately 5.891 seconds. Physical cleanup completed at approximately 93.406 seconds, followed by a stable window and independent exact Deleted/physical verification. There was no retry/finally DELETE and no Deleted-with-residue sample in this ordering.
    3. After actual create expiry, exactly one public DELETE also led to eventual cleanup, with no remaining empty runc state directory, manual rmdir, or service restart. However, this ordering reproduces the remaining gap below.

    The previous campaign separately demonstrated a cold create succeeding after approximately 210.810 seconds with the unchanged ordinary 30-second RPC configuration. That success was not rerun in this follow-up.

    Remaining reproducible gap: expire first, then delete once

    Native request: POST /api/environment/v1/environments, unique name/request ID, runtime=runc, immutable image, inheritEntrypoint=false, failover=false, create/schedule/init budgets 35/1/1 seconds. Once the server response and Failed state were observed while preparation resources remained, send one DELETE /api/environment/v1/environments/{id}. Disable client retries/redirects; observe without any further mutating request.

    Exact environment: default-macos-sdk-af43f22dae6247ce8a66c2b45272d5fe; generation 1; create request ID create-5efdfc25-a941-4750-9ca8-4fdb20a677df.

    Times below are relative to test start; POST was issued at approximately 0.168 seconds.

    Event Elapsed seconds
    Create returns HTTP 503, structured OUTCOME_UNKNOWN/Timeout expired 35.169
    The sole public DELETE is sent 36.170
    DELETE returns HTTP 200 36.212
    First exact Deleted receipt, public instance 404 and empty full inventory/queue, but netns still exists 37.298
    Last sampled Deleted/netns overlap 101.388
    First independently observed physical all-empty state 102.509
    Stable observation completes 132.660

    There are 58 samples with the exact generation-1 Deleted receipt and resources_held=false, empty inventory/queue and empty sbox/runc lists, but netns runc-sbox-bebd970e-bc35-4375-b85f-2f3c2c03dabb still present. Receipt result digest: 9dfbd7968cec26755acb3dcd69f3263605f7ada83ae62c6b5041f3e4d02ce39b. Processes, runtime mounts, state/bundle/private-storage paths and runc cgroups were empty in those samples. Final independent verification confirmed that the netns also disappeared. Observation DELETE count 1; post-observation catalog-retirement DELETE count 0.

    Thus ACK precedes observed physical completion by approximately 66.297 seconds, and the first observed Deleted receipt precedes it by approximately 65.211 seconds. This is eventual cleanup, not a permanent orphan. resources_held=false is a scheduler-reservation fact and is not, by itself, a physical-cleanup completion guarantee. The internal Deleted receipt is also not a separate public cleanup-complete operation.

    Suspected boundary, not a proven unique root cause

    At the exact packaged ADX source:

    This path is consistent with the observed ordering, but the current trace does not correlate every internal RPC/preparation/rollback boundary and does not prove it is the sole cause. It is also consistent with the existing logical-deletion limitation already acknowledged in the ce80a189 delivery comment, not a newly attributed regression in sandboxd #74.

    Suggested remaining work and acceptance

    • Use a controlled slow-import fixture and correlate the same operation/request, environment, runtime and generation across RPC completion, underlying import completion, rollback completion and Deleted publication.
    • Either fence Deleted on completed physical cleanup, or explicitly preserve logical deletion and expose a separately queryable cleanup pending/complete/failed state with durable ownership/identity. Whichever contract is chosen, its completion signal must agree with independent netns/mount/process/state/bundle/private-storage/cgroup observations. An empty published-runtime list cannot by itself fence unpublished in-flight preparation.
    • Regress both early-delete and deadline-expiry-then-single-delete, require no additional public DELETE, eventual termination, no observed revival, no held resources or residual empty runc directory, and compare the advertised completion signal against physical release. Immediate interruption of image import is not required for eventual-cleanup acceptance.

    This follow-up is local native-runc deployment evidence, separate from remote CI and author component tests. Default Standalone E2E does not enable runc or establish this contract. Sampling was approximately once per second and does not exclude every sub-second transition. KVM/QEMU/macOS guest acceptance was not rerun here. Tool assertion/preparation failures and raw traces are retained locally, not uploaded; the table above is a sanitized excerpt. All four owned fixtures/networks were removed only after physical verification, with no task loop device, listener or allocation journal remaining; existing services, source VM and source disks were protected. No product code, image publication or cloud resource was changed.

    Recommendation: keep #96 OPEN for this remaining contract/observability item. If it is split into a linked follow-up, explicitly agree on that scope before closing this tracker; final eventual cleanup or Delete ACK alone does not establish the currently requested full acceptance.

  5. rcyw commented on Oct 15, 2026

    @rcyw
    CollaboratorAuthor

    Controlled-import follow-up to the previous deployment evidence: the previously suspected boundary is now reproducible with correlated stages. For this path, client Start cancellation and Deleted publication happen while the native Start handler is still blocked in image preparation; rollback completes only after the layer body is released. This establishes a concrete mechanism for the remaining gap, not the unique cause of every possible failure. Recommendation: keep #96 OPEN for the completion/ownership contract.

    @WenYuLuo, the following evidence and proposed contract may help scope the next component fix. The existing deadline and late-runc-start fixes remain effective; this is not attributed as a new regression in sandboxd #72/#74.

    Inputs and reproduction

    • AKernel 9f24718638131b09ad03a274b9ba52ff76396a38, sandboxd d5886b42ceff89bf0bf9baf7f73f5c534f689dee, ADX platform/Execd ce80a189f352f1ee2ebb6a9f30ee218f2ae896d3. These are the actual integrated inputs from the previous comment, including the packaged ADX fix, not a gitlink-only candidate.
    • Same audited immutable runc-enabled image ID sha256:5d25768d05ca714803b95c685a915e0394a9b51998e88e401bac27f0c869fc2e. Each node rechecked the actual sandboxd binary SHA-256: 733780c1a72549e89b4e4206f3068e52375d22f658e575df69fca460ddbe7207. This phase reused the image; it did not rebuild it or rehash the entire payload again.
    • Two fresh, exclusively owned nodes, separate bridges/data/config, verified empty OCI caches, native runc enabled, netns cache disabled. The diagnostic image contains one text file in a 10 KiB tar layer, with a unique immutable manifest/layer per run. A read-only local HTTPS registry sends correct headers/digests but gates the layer body; its 180-second safety limit was not reached. No real registry credentials or guest image are needed.
    • Keep ADX deadline defaults. Only redirect adxlet.sandboxd_socket through a transparent, byte-for-byte UDS observation proxy. Use the native API with retries/redirects disabled, HTTP operation timeout 300 seconds, and create/schedule/init budgets 35/1/1 seconds, runtime=runc, inheritEntrypoint=false, failover=false. No product source or binary was modified.

    Send one POST /api/environment/v1/environments with a unique name/request ID. After the real server deadline response and Failed state are observed with the layer still blocked and the preparation netns present, send exactly one DELETE /api/environment/v1/environments/{id}. After observing the matching generation's Deleted receipt, public 404 and empty complete inventory/queue, keep the body blocked for another 15 seconds (A) or 30 seconds (B), then release it. Make no further public mutating requests. Observe physical cleanup for at least 30 stable seconds and verify the exact terminal identity independently.

    The create request/operation ID is associated with the environment/runtime/generation through the API record and the single observed native Start; DELETE has its own request ID on that same execution.

    Correlation A B
    Create operation/request ID create-phase-8cf2a0b3-32bc-4483-b2ce-8f57c62ac648 create-phase-8c1ba383-792f-4b99-9b4e-81f31a02d223
    Environment ID default-macos-sdk-a424223c4de0498dbcdd03095e179be5 default-macos-sdk-832f90a923ce4609abfb3cd87dc133fb
    Runtime ID / generation Environment ID + -1 / 1 Environment ID + -1 / 1
    Native Start trace ID e8a2dfcbddac3bf71cb42d08fde9d617 b2e37fcb885c8f79c5c542af2059dafb
    Native Start count / native Delete count 1 / 0 1 / 0
    Public create / DELETE count 1 / 1 1 / 1

    Observed order

    All times below are seconds from public POST begin, unlike the previous comment's driver-start origin. Receipt writes use Redis MONITOR server timestamps; import/rollback use the native log timestamps; syscalls use the host clock. The native handler interval is bracketed by successful metrics polls, not inferred from the time a log watcher read the line.

    Stage A: hold 15 seconds B: hold 30 seconds
    Client Start RST_STREAM, code 8 35.001 35.002
    Create response: HTTP 503, matching OUTCOME_UNKNOWN / unknown / Timeout expired 35.002 35.003
    Failed committed, revision 2 / resources_held=false 35.002 35.002
    Matching Deleted receipt written 35.099 35.194
    Sole public DELETE ACK, HTTP 200 35.140 35.233
    Registry actually releases layer body 50.646 65.637
    OCI import/mount succeeds 50.651 65.642
    Runtime Start rejected with context canceled; filesystem rollback begins 50.653 65.644
    OCI unmount success 50.654 65.646
    Exact netns umount succeeds, then its path is unlinked 50.681 65.687
    Native Start handler completion interval (50.635, 50.688] (65.659, 65.713]
    First stable all-empty physical sample 50.985 65.978
    Stable observation completed 81.427 96.381

    There are 43 A samples (35.459–50.609 seconds) and 84 B samples (35.563–65.605 seconds) with public 404, empty full inventory/queue, the exact Deleted receipt, empty ADX held resources and resources_held=false, while the layer body is still unsent, native Start completion count is zero, and the preparation netns remains. Sampling was approximately every 0.35–0.38 seconds; RPC parser errors and metrics probe errors during both runs were zero.

    The netns identities were A runc-sbox-f8aec300-4c3d-4b9e-82ca-a4e0decbd155 and B runc-sbox-03a045d3-c2bd-4d46-94f9-7ea0a99ed636. Saved syscall traces show umount2(..., MNT_DETACH) = 0 followed by unlinkat(..., 0) = 0 for those exact paths, after layer release. MNT_DETACH alone is not the whole release proof; it is combined with handler completion and the subsequent stable independent physical probes.

    B's RPC cancellation moved only 0.0005 seconds, and Deleted only 0.095 seconds, relative to A. Layer release moved 14.990 seconds and first physical all-empty moved 14.993 seconds. Cleanup follows the controlled import release, rather than a fixed delay after Deleted. The same native trace ID connects each Start receive and its later context-canceled failure. No runc workload/shim/QEMU process was launched, and no residual empty runc state directory was found.

    Concrete source boundary

    1. Native preparation remains in flight after transport cancellation. server.go:1392–1402 starts filesystem/resource preparation and waits for both; fsmanager.go:106–107 has no caller context. MountOCI:1023–1025 uses an HTTP Post without that context, and the image worker uses context.Background at line 78. The path does not wait for a canceled Start's rollback before the client RPC ends.
    2. ADX's execute_start:310–317 sets Failed before the RPC; a transport error leaves that state. remove:1152–1197 guards Uncertain, but Failed can return Ok when no physical mapping or published runtime list entry is found. An empty published-runtime list does not include this unpublished preparation allocation. The observed native Delete count of zero is consistent with this path.
    3. Controller::delete:958–972 publishes Removed/Deleted after cleanup returns Ok. It therefore precedes the in-flight native operation's completion in this experiment.
    4. After body release, the existing runc Start cancellation gate:169–174 correctly rejects the late result; server.go:1373–1388 performs deferred rollback/resource release. The Start result counter:36–42 increments only after the handler, including its defers, returns. Its poll interval is a completion bound, not an exact timestamp within that interval.

    Together, the controlled intervention and fixed source establish this path: client RPC canceled → Failed → no published runtime interpreted as successful removal → Deleted → import finishes → canceled runtime launch rejected → deferred rollback/resources released → native handler completes. This combines a cancellation-propagation gap with a completion-confirmation gap; it is not evidence of a permanent orphan, nor proof that sandboxd alone explains every case.

    Suggested fix and acceptance

    • Define the completion contract first. If Deleted means physically cleaned up, wait for a generation-specific native operation/cleanup fence that confirms preparation has stopped allocating and rollback has completed. If logical deletion intentionally precedes cleanup, expose a separately queryable cleanup pending/complete/failed state, retain ownership and failure evidence, and ensure the complete signal agrees with independent physical observations. resources_held=false remains a scheduler-reservation fact.
    • Add native in-flight Start tracking and query/wait/cancel semantics even before a runtime is published. Propagate cancellation through Prepare, image-manager requests, registry reads and extraction workers, while respecting shared-layer consumer/refcount ownership. Faster cancellation alone does not make transport completion a rollback-completion acknowledgement. Cleanup failures must remain observable/retryable with ownership retained.
    • ADX should distinguish transport failure from confirmed native completion and consult that fence before treating an empty runtime list as cleanup success. Do not just replace Failed with Uncertain: the current Uncertain+empty-list guard still lacks a completion witness for a Start that never publishes a runtime and may never converge. Keep background reconciliation and prevent old-generation results from reviving or overwriting a new execution.
    • Regress the two controlled hold durations above, plus the existing early-delete ordering. Require one create/one public DELETE, no extra delete after recovery, exactly one native Start, no late Running/workload, eventual physical all-empty and at least 30 stable seconds. During the blocked interval, either withhold the physical-complete signal or expose pending and retain the operation's cleanup ownership. Add separate failure/recovery cases for permanent import errors, rollback failure, shared-layer cancellation, service/worker restart, same-operation replay and generation isolation; these additional cases are recommendations, not passed tests.
    • Integrate any component fix into the actual AKernel build inputs, including matching ADX platform/Execd archives and checksums where applicable, then rerun native-runc deployment acceptance. A source pointer or default CI/Standalone E2E result is insufficient for this path.

    Both diagnostic runs actually completed locally; this is separate from remote CI and author reports. The harness's 7 offline tests passed, but they do not establish product acceptance. Full large-image successful cold import, normal runc smoke, real KVM/QEMU/macOS guest acceptance, and restart/failure-recovery scenarios were not rerun in this controlled phase. Earlier passed layers retain their original version/scope.

    Sanitized times/identities above are the evidence excerpt. Raw traces, configs, frozen source/hashes and a 54-file SHA-256 manifest remain local under artifacts/experiments/2026-10-15-issue96-phase-correlation/ (A a-hold15, B b-hold30); they are not uploaded or backed up off-host. The diagnostic harness and durable result are saved as uncommitted reviewable workspace changes, not yet a published product fix. The initial trace-ID parser returned null; a separate pre-teardown audit extracted only native line/time/TRACEID and confirmed the matching receive/failure IDs. Preparation/tool failures were retained separately, not counted as lifecycle success.

    After independent exact-generation terminal/inventory/physical rechecks, both owned nodes/bridges were removed and their tracers, ports and loop devices released, with zero additional public DELETEs. Existing services/source VM/disks were protected. No service restart/stop was used to obtain cleanup success; owned-node teardown occurred only after the stable physical proof. No product code, image publication or cloud resource was changed. The remaining completion/ownership contract is still unimplemented and unaccepted; eventual cleanup and Delete ACK alone are not grounds to close #96.

  6. WenYuLuo commented on Oct 15, 2026

    @WenYuLuo
    Collaborator

    Follow-up to the controlled-import diagnosis: the timeout reporting and OCI cancellation-propagation fixes are now submitted, with component regression and isolated native-runc acceptance completed. Keep #96 open for AKernel artifact integration and the remaining agreed acceptance scope.

    • ADX !51, targeting GitCode master, head 9ce1e4ab57689ac6ee71669af067349dd2058030: confirmed create deadlines return HTTP 504 / DEADLINE_EXCEEDED, with the active phase in message, for example environment initialization deadline exceeded during backend preparation/runtime start. The phase is retained on failure replay. The original absolute execution deadline is preserved; the bounded response-reporting allowance does not extend the execution budget. Confirmed terminal failures are not automatically retried; genuinely unconfirmed publication or lost replies retain their appropriate same-operation/unknown-outcome handling.
    • sandboxd #75, targeting main, head b56f5ed425fab2ef0d4d9ae890ee95d67d000086: propagate the Start context through filesystem/rootfs preparation, OCI rootfs and image-volume mounting, inherited process metadata lookup, the image-service facade and HTTP mount endpoint. The registry client now passes remote.WithContext(ctx) to the registry library; previously its ctx was only used for tracing. Layer workers and shared-resource waits also honor cancellation, while preparation results are collected before rollback releases ownership. Cleanup runs independently of the cancelled Start context. Live shared consumers remain usable; cancelling one HTTP/2 request no longer closes unrelated streams on that connection. This covers regular OCI preparation; S3/FUSE daemon startup and Nydus bootstrap retain their existing behavior.

    The reproduced mechanism is therefore addressed at both boundaries: a confirmed deadline identifies the failing phase, and cancellation interrupts blocked OCI reads instead of waiting for the registry body to be released. Existing logical deletion and eventual reconciliation semantics are retained.

    Inputs actually exercised

    AKernel 9f24718638131b09ad03a274b9ba52ff76396a38, real Redis/API Server/Coordinator/adxlet, native runc 1.5.1 enabled, and the rebuilt sandboxd #75 binary. ADX acceptance used rebuilt binaries containing the !51 timeout fix from its pre-retarget source (4e6abe2 plus a452f65); affected production source hashes were verified to match the patch retargeted to master. These are task-built binaries, not a newly published ADX archive integration.

    Final isolated image ID: sha256:283ac86ba06b6f7c133744edd8077296e1ea76e97b74d46c13513e19063c29b0. The running sandboxd binary was verified against the built SHA-256: 84f67f3c56d9318d2422e37f940c8546c8c6afcb0491f2945703515692d0cad8.

    The controlled registry serves valid, distinct OCI digests. Native runc is enabled and netns caching disabled. Create/schedule/init budgets are 35/1/1 seconds for expiry cases; the successful control uses a 10-second init budget. The registry flushes response headers and withholds the layer body. There is no elevated 900-second RPC workaround.

    Scenario Create duration / result DELETE duration / HTTP First all-empty sample after create response
    Successful cold-create control 1.143 s / HTTP 200 0.176 s / 200 Verified after deletion
    Deadline expiry, then delete 35.004 s / HTTP 504 0.005 s / 200 0.150 s
    DELETE queued during blocked create 35.005 s / HTTP 504 32.845 s / 200 0.171 s
    SSE deadline expiry 35.006 s / HTTP 200, final errorCode=504 0.004 s / 200 0.132 s

    All three expiry cases report DEADLINE_EXCEEDED / never / terminal with the backend preparation/runtime start message. Their registry readers disconnect on cancellation; the layer bodies are never released. Independent sbox/runc lists, netns, runtime mount/state paths and process observations become empty and remain stable for more than 30 seconds. No late Running is observed. There is exactly one public DELETE per case, no additional cleanup DELETE, manual rmdir, or service restart to obtain cleanup success. The task container is removed after verification and the pre-existing node remains running. The expiry case also checks failed-create message replay.

    The early DELETE still waits behind the serialized create until its deadline; that explains the 32.845 seconds above. These fixes do not change lifecycle queue ordering.

    Regression gates

    • ADX: 372 non-ignored component tests, all 30 real Redis/network/mTLS control-RPC tests plus the background-trace test, SDK transport/create checks, make rust-check, documentation and diff checks pass.
    • sandboxd: make check-fmt, make vet, make test pass (41 packages pass, zero fail; eight packages have no tests). Cancellation/shared-consumer regressions in all five affected packages pass five repetitions under go test -race, including blocked manifest/config/layer reads, Start rollback, attached OCI volumes, cancelled shared-rootfs creators/waiters, shared-layer contention and actual HTTP/2 stream isolation. The blocked-download and HTTP/2 tests both demonstrated the failures before their fixes.
    • make release-binary succeeds in the Ubuntu 20.04 build image with seccomp/apparmor tags; the final binary is the one used for the four native-runc cases above.

    This evidence covers the controlled OCI cancellation path on the dedicated development host, separately from CI and published-release acceptance. AKernel's actual release/archive/checksum inputs still need promotion and integration after review. Full large-image successful import, KVM/QEMU/macOS guest acceptance and restart/rollback-failure recovery were not rerun in this campaign. No physical-cleanup acknowledgement contract is inferred from a DELETE ACK.

    Sanitized results are preserved locally in output/issue96-cancel-native-summary.json; the development-host log is /var/log/sandboxd-issue96-cancellation/native-final-3.log, with the fixture and raw evidence under /root/sandboxd-worktrees/start-preparation-cancellation-issue96/output/issue96-cancel-native-final/. These paths are retained evidence, not public artifact downloads.

  7. WenYuLuo commented on Oct 15, 2026

    @WenYuLuo
    Collaborator

    Follow-up: the early backend timeout chain and non-timeout error classification are now fixed in the existing PRs, with component checks and real public-API standalone regression completed.

    • ADX !51, targeting GitCode master, head 17d4833160e507e7352b2ac430c368a142042f6a (production fix 075af5c, followed by E2E-driver checks).
    • sandboxd #75, targeting main, head 02386fea2050c4a77cfaa9df5edfe72fd35fc987.

    Timeout propagation

    sandboxd now preserves typed network/image-pull and runtime-start deadlines, including an internal timer that expires while the overall Start/create budget still has time remaining. The error message retains the concrete stage and original cause. runsc RPC timer errors and the runc startup-wait timer retain context.DeadlineExceeded; filesystem preparation labels rootfs/image-volume pulls and inherited process-metadata lookup separately.

    ADX preserves that application timeout through adxlet, Coordinator and API Server. A confirmed, durably published timeout returns DEADLINE_EXCEEDED / never / terminal, with HTTP 504 for JSON or errorCode=504 in the SSE final event. Both same-request replay and authoritative failed-record replay retain the exact message. The absolute execution deadline is unchanged; uncertain transport/lost-reply or unconfirmed-publication results retain their existing retry/outcome contract.

    Actual message excerpts from the native run:

    • Image: sandboxd: filesystem preparation: image pull (rootfs): context deadline exceeded: ... net/http: TLS handshake timeout.
    • Runtime: sandboxd: runtime start (runc): waiting for runc container start after 10s: context deadline exceeded.

    The internal sandboxd-start-failure trailer identifies an application deadline reply; it is not a physical-cleanup completion acknowledgement. Cleanup continues through rollback/reconciliation independently of the cancelled request context.

    Non-timeout errors

    Other backend gRPC errors retain their category and original message instead of being flattened to Unavailable or inferred to be a timeout. Explicit cancellation takes precedence over network-timeout normalization, and actual connection failures remain transport failures. Classification uses typed errors/status codes.

    The real registry controls verify both JSON and SSE:

    Registry response Public error code Retry / outcome
    401 UNAUTHENTICATED never / not_started
    403 PERMISSION_DENIED never / not_started
    404 NOT_FOUND never / terminal
    429 RESOURCE_EXHAUSTED never / terminal

    Each response retains the original registry rejection message. Backend registry 429 is distinguished from node admission/capacity rejection: Coordinator returns the backend failure directly; node admission retains after_backoff / not_started and its existing retry behavior.

    Regression results

    The final run uses AKernel 9f24718638131b09ad03a274b9ba52ff76396a38 as an isolated standalone fixture with real Redis, API Server, Coordinator, adxlet, rebuilt sandboxd and native runc. These are public ADX API calls in that fixture, with SDK regression checked separately.

    Internal failure JSON response time SSE final time Overall create budget
    Image TLS handshake timeout 102.885 s 102.531 s 180 s
    runc startup timeout (10-second internal timer) 11.049 s 10.216 s 180 s

    All four internal-timeout cases return the precise stage and DEADLINE_EXCEEDED / never / terminal before the total deadline. SSE transport is HTTP 200 with final errorCode=504. Exact-message replay passes for the original Request ID and a new Request ID resolving the same failed Environment. The TLS timings above belong to this controlled blackhole/retry fixture, not a universal image-pull timeout setting.

    All eight non-timeout registry controls pass. Healthy create/delete succeeds before and after fault injection. Final independent sbox/runc inventory, netns/state-directory, mount and runtime-process audits are empty. The disposable container is removed and the pre-existing node remains running; running binaries are checked against the fixture artifacts. The tested sandboxd SHA-256 is eb71d87a96e481b23761d2865b2624d43ca713a9e15105067248a186b5d104cb.

    Additional gates:

    • 73 adxlet lifecycle/backend RPC tests and eight API error tests pass, including preservation of all 16 non-OK backend gRPC categories and a real TCP connection-failure control.
    • All 30 real Redis/network/mTLS control-RPC tests plus the background-trace test pass, including real API Server HTTPS fixtures. All 294 Environment SDK tests and focused JSON/SSE transport checks pass.
    • Full-workspace/all-target/all-feature Clippy, Rust formatting, three release-binary builds, documentation and diff checks pass. The E2E driver's validation remains active under Python -O; the corrected driver passes the complete native run above.
    • sandboxd Linux format/vet/full-unit and targeted race checks pass. GitHub CI run 38745938134 passes all 14 jobs at 02386fe, including all seven runtime E2E variants. The early internal runtime-timeout fixture above specifically exercises native runc; runsc's internal timer also has a component regression.
    • ADX !51 static checking, Ruff, forbidden-word scanning, anti-poisoning and open-source compliance all report SUCCESS at 17d4833 (bot result 194038829, 2026-10-15 22:22 +08:00).

    Final native evidence is retained on the development host under /root/adx-issue96-timeout-20261015/out/internal-timeout-native-codecheck/; log: /var/log/adx-internal-timeout-native-codecheck-20261015.log. These are retained local evidence paths, not public downloads.

    Keep #96 open for reviewed artifact promotion/AKernel release integration and the agreed remaining acceptance scope. This follow-up records source-built standalone acceptance; it does not establish published-release or cluster deployment acceptance.

  8. rcyw commented on Oct 16, 2026

    @rcyw
    CollaboratorAuthor

    Post-merge deployment acceptance completed on 2026-10-16 (Asia/Shanghai). The original single-node cold-import/deadline/cancellation report is covered on the fixed AKernel PR #100 squash below. Closing #96 as completed under the eventual physical cleanup contract, with the limits and original harness failures recorded here.

    This supersedes my earlier suggestion that logical Deleted must wait for physical release or require a new public cleanup API. Logical deletion/scheduler release and physical reclamation remain separate; background rollback/reconciliation must eventually release the exact allocation without another public DELETE or a service restart. Delete ACK, logical Deleted, empty inventory, and completed physical cleanup were observed separately. Their timing need not be strongly consistent.

    Actual integrated inputs

    • AKernel PR build(runtime): refresh ADX rc1 and sandboxd Start fixes #100 squash: b5db829f9da17c970d4498d460f235fbdb8cea1c; source-installed SDK from its exact sdk/python, version string 0.1.6rc9, actual backend adx-environment==0.1.0rc1 (no adx-sandbox).
    • sandboxd gitlink: 02386fea2050c4a77cfaa9df5edfe72fd35fc987. It is obtainable from the canonical source and has the same tracked tree (716a846fcc3025fb739c7c6add3170bc97b99449) as merged sandboxd build: package the complete gVisor akernel.3 runtime bundle #75 squash ff3c4d82b31b0ebd3ed99fa436816c3cbc28e292.
    • ADX platform/Execd source: af7d82f04a52d4604ea7ca4c550f09c6114dee4d. Actual platform archive SHA-256 84f2cc11dca67aed642f4908c0e74ba1978d272491e239adad2ad03ba302da77; Execd archive 62edc1079f5fcb500d25f397bf6c43bc7ff74b885ec8ec386069df138c2c8db1. Archive manifests, delivered node binaries, and Execd inside the runtime rootfs were checked, not just the source gitlink.
    • Newly source-built linux/amd64 node image: sha256:dd88bef66249d0a9bb8f752e30a6a16a87aa0dbb9eab044e377bd5b89ca32a7c; native runc 1.5.1 included and enabled, with the shim/helpers present. Kata and Firecracker were disabled. sandboxd was compiled from the fixed checkout; the build excludes .git, so an embedded sandboxd VCS revision is not claimed.
    • Cold runner: immutable digest sha256:b584e2bc6173fcd065df33d15fba050473ab07289be284d061d323cd467e4d8b. Each cancellation fixture used its own initially empty OCI cache, node, data directory, network and exact ownership checks. ADX RPC defaults were retained; no 900-second workaround.

    This round's actual runtime evidence

    1. A real 15-layer cold import succeeded in 86.991 seconds, with one create operation and one native Start, using a 660-second create / 600-second scheduling budget. Command execution and independent deletion passed. No entrypoint/QEMU was started in this import-only case. Its maintained tool includes final idempotent deletes, so it is not the strict one-DELETE proof.
    2. Four controlled slow-import cases used unique valid OCI layers, create/schedule/init budgets 35/1/1 seconds, runtime=runc, no inherited entrypoint, no transport retries, and a layer body that was never released. All returned the original request ID with DEADLINE_EXCEEDED / never / terminal; the registry reader disconnected at about 35 seconds. Exactly one native Start was observed per case, with no late runtime and at least 30 stable seconds after cleanup.

    Times in the following table are seconds from public POST begin. The physical column is a successful probe's sampling interval, not an exact release timestamp.

    Case Create result / seconds Public DELETEs during automatic observation First settled all-empty physical probe
    Deadline, no delete during observation HTTP 504 / 35.004 0 35.215–35.338
    Delete during blocked import HTTP 504 / 35.020 1 35.114–35.233
    Delete after deadline HTTP 504 / 35.003 1 35.435–35.556
    SSE deadline, no delete during observation HTTP 200, final errorCode=504 / 35.003 0 35.316–35.425

    The early DELETE was sent at 0.499 seconds and acknowledged at 35.062 seconds: it still queues behind the create deadline, so immediate cancellation is not claimed. Native handler-completion poll bounds for the four cases were respectively (34.9976,35.0496], (34.9881,35.0399], (35.0191,35.0711], (35.0119,35.0638]. Unlike the prior reproduction, handler completion/rollback no longer waited for manual layer-body release. For the two zero-DELETE cases, Failed catalog plus automatic physical cleanup was frozen first; explicit catalog retirement happened afterwards and is not used to prove automatic cleanup. The one-DELETE cases needed no subsequent public mutation.

    1. A separate real 15-layer empty-cache import cancellation sent exactly one POST and one DELETE. The create returned HTTP 504 / DEADLINE_EXCEEDED / never / terminal after 35.0026 seconds. DELETE was sent 5.714 seconds after POST begin (6.032 on the driver clock), and returned 200 at 35.046 seconds after POST begin. Actual native labels bound request create-73e75fda-cfcc-4d5a-a97d-1a0dea1c6a50 to environment default-macos-sdk-4be916b5bdc6497fab05b6e9f16fb646, runtime environment ID + -1, generation 1; 32 assignment samples and Start counter 0→1 agreed. All 15 layers were cache misses and cancellation was logged. The exact Deleted receipt, public 404, complete inventory/queue, held resources and all physical items were empty for 30.297 seconds / 28 samples, without another DELETE or restart.

      This runtime harness originally failed after making those observations: it incorrectly required an already published control.runtime during canceled preparation. The original failed run, summary, journal, logs and script are preserved. A separate read-only reassessment passed using the actual native Start labels/assignment/counter; it made zero new public mutations. Three validator regressions passed, accepting the complete unpublished-preparation identity and rejecting wrong environment/generation or missing runtime labels. Netns ownership is scoped by an initially empty exclusive node, its sole Start/assignment and sole preparation netns; a direct netns-to-label field is not claimed.

    2. The maintained opt-in fault-runc-delete-retry test actually ran: one passed, zero skipped. One public DELETE was followed by automatic reconciliation in about 0.406 seconds. Metadata, private-file content, runtime state, mounts, netns/cgroup, assignment and ADX held resources were retained at the injected failure and before the successful native retry. Init naturally exited after TERM; continuous process survival is not asserted. Catalog cleanup was recorded separately.

    3. Ordinary native runc command/stdout/stderr/nonzero exit and binary-file transfer passed; a real KVM ioctl returned KVM_API=12 VM_CREATED=1. Invalid snapshot input was rejected before QEMU, followed by independent physical cleanup. Service TLS trusted/untrusted/wrong-host checks passed across control/resources, command, files, real command-watch handshake, PTY and CLI inventory/deletion.

    4. Two fresh, serial-use private-baseline macOS guest lifecycles passed trusted SSH and host-key checks, Darwin / macOS 15.8 / 24H23 / admin GUI, Web/RFB and six complete 1920×1080 PNGs, command streams/nonzero status, binary transfer, and actual mixed-case desktop positive/negative tasks. Both live QEMUs were correlated to the exact environment/generation/runc cgroup and held KVM VM/vCPU FDs. Both shut down normally and passed independent deletion, forced=false. The 28 adapter runtime/tool/dependency hashes stayed unchanged through both runs.

    5. Controlled loss of one real successful create result retried the same operation/name, exposed one visible instance and incremented native Start only once. This is a controlled discarded return value, not a real TCP-failure test. Offline adapter regression: 82 passed, one skipped for missing host zsh.

    Independent physical checks required empty sandboxd/runc inventory, no runtime/QEMU/shim processes, netns, private mounts, runtime state, bundle directories, private storage or runc cgroup leaves. Exact-generation terminal receipts and ADX held resources were checked separately from those physical observations.

    Evidence retention, failures and final resources

    Sanitized results are included above. Complete raw evidence and diagnostic sources remain local only, not publicly uploaded or backed up under artifacts/experiments/2026-10-16-issue96-post-pr100/. The summary is validation-summary.json, SHA-256 b474ba737a78f7f16a7f00f602608f90499b734fbeabfcaf915fcc7f6d69e1ae; its 24 evidence-file hashes were rechecked before posting. Main evidence entries:

    • controlled-four-case-results.json and per-case frozen fixture sources in controlled-case-sources/ (17 tool regressions passed).
    • Real cancellation original failed run 20261016T131641068305-issue96-real-cancel-cold-single-delete-a8d89798df51; separate read-only reassessment 20261016T132251766398-issue96-real-cancel-evidence-reassessment-9fd27e8d2eda.
    • Guest runs 20261016T130257724036-issue96-post-pr100-authenticated-01-4fe810107b38 and 20261016T131436427284-issue96-post-pr100-authenticated-02-5f882a0400b1.
    • Final physical/protection audit 20261016T132702011544-issue96-post-pr100-protection-final-0e42f286db1e; evidence/PNG/provenance summary audit 20261016T132710841384-issue96-post-pr100-acceptance-summary-c950499aa3b7.

    A real-cancellation retirement wrapper also exited 1 on Docker “not found” diagnostic-text matching after its strict helper successfully removed the exact owned node/network. That failed run remains unchanged; a separate read-only final audit confirmed both IDs absent, ports refused and owned loops zero. Initial download-routing and overstrict empty-cache preflight failures are retained separately and are not counted as product passes. No failed run has been relabeled as successful.

    All 7 task nodes and 7 networks were removed after physical proof; all 14 ports are closed, owned loop residue is zero, no host QEMU/runc-shim remains, and the private-baseline lock is released. The seven pre-existing containers including the source VM retained their identity/image/state/start/finish data. Original disk file-set/inode/device/size/mtime remained unchanged; this is a metadata check, not a new byte hash of the original 128-GiB disk. Private local inputs/images/evidence remain retained. No shared service was restarted/stopped, no source VM/disk was modified, no image was published and no cloud resource was changed.

    Closure scope / remaining independent work

    The original single-node cold-import budget, real deadline expiry, blocked OCI cancellation and eventual physical reclamation are accepted for this fixed local build. This does not establish synchronous physical release at Deleted or a new public cleanup-complete guarantee.

    Not run in this round: daemon/control-plane restart, real unmount/rollback failure recovery, distributed clock skew, multi-node/multi-instance or shared-consumer stress, real TCP create-result loss, pause/resume, Kata/Firecracker/GPU, Nydus/S3/FUSE paths, cloud deployment, or acceptance of newly published release images/wheels. The author's early internal-timeout and non-timeout registry-error matrix remains separately attributed author evidence; it was not independently rerun as part of this campaign. Default CI/Standalone E2E is not used as a substitute for any native/opt-in/guest acceptance above. SDK security-default tracking in #97 and the native registry's existing skip_verify behavior remain separate from this closure; the adapter explicitly enables service verification.

    These broader scenarios can be tracked independently without extending this original report indefinitely. The historical observations and earlier open recommendations remain preserved; this comment records the accepted design and post-merge result used to close it.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions