Identify intermittent Nextest capture leak in S3 tests

[gaborbernat]

Problem

The combined storage validation command can exit an S3 integration test with code 0 while Nextest still reports the
test process's captured stdout or stderr as open:

cargo nextest run --package peryx-storage --package peryx-ha-distributed --all-features --profile ci

The original run at f78ba1d7 completed 2,095 tests: 2,093 passed and two failed because of leaked handles. A diagnostic
run at the same commit reproduced LEAK-FAIL in a third S3 test while the original two passed. The affected test is
therefore not fixed to one multipart scenario.

Observed cases

• test_s3_cleans_up_when_peer_completion_status_cannot_be_checked
• test_s3_drains_started_parts_before_aborting
• test_s3_reports_an_unreadable_multipart_journal

Each stdout block ends with test ... ok, followed by test failed: exited with code 0, but leaked handles.

Diagnosis

Nextest 0.9.143 detects a leak
when the test process has exited but its captured stdout or stderr has not reached EOF within the configured second.
The count and executable hashes stayed stable across the diagnostic runs, which rules out stale shared-target binaries.

The S3 fixture cannot retain those Nextest pipes. The test launches it with Command::output; Tokio replaces
the fixture's stdout and stderr with private pipes
and waits for exit and EOF on both pipes.
Sampled passing runs found no reparented fixture descendant. The remaining diagnosis must identify every process or
duplicate file descriptor that retains the test process's actual Nextest pipe after exit.

A detached f78ba1d7 reuse run enumerated the same three executables and 2,095 tests without invoking Cargo's build
path. The HA and integration executables matched their original hashes. The shared target had overwritten the storage
library and S3 fixture, so those two files were rebuilt from the detached source but were not byte-identical to the
original run.

A temporary Nextest 0.9.143 diagnostic build emitted the runner's existing time_to_close_fds_nanos value without
changing leak detection. A low-overhead full run performed one relaxed atomic maximum update per test and
reported a suite maximum of 7,542 ns against the 1,000,000,000 ns limit. It passed all 2,095 tests and produced no
timeout event. The sampler captured 63 S3 test processes; none held a duplicate of its stdout or stderr pipe endpoint.

The passing run does not identify the historical owner. macOS reused pipe addresses for later processes, and Nextest
kept some read descriptors open after observing EOF, so neither an address match nor reader lifetime proves a retained
writer. The missing evidence is a failing timeout record containing the test identity, Nextest's reader descriptors,
and a simultaneous process-wide descriptor snapshot.

Required change

Capture the retaining PID and file descriptor during a failing run, then repair the owner that created it. Do not
change S3 fixture lifecycle code unless the capture proves that it owns the Nextest pipe.

Do not increase Nextest's leak timeout, exclude tests, disable leak detection, retry jobs, or add CI cleanup. Preserve
the multipart cleanup assertions and failure scenarios.

Acceptance checks

• A failing diagnostic names the process and file descriptor that retains the test's Nextest capture pipe.
• The owner closes and joins that resource before the test process exits.
• The combined command completes all three observed tests without LEAK-FAIL.
• The completion-status failure still aborts the orphaned multipart upload and removes its journal.
• The concurrent-part case waits for started parts before sending one abort request.
• A regression fails when the repaired owner omits shutdown or join.

Architecture and test boundary

Keep S3 lifecycle behavior and its integration tests in peryx-storage. Repair the narrow owner that creates the
retained handle; do not add CI-specific cleanup logic or a repository script.

[gaborbernat]

Another test process retains the handle; neither S3 test does. On macOS, Rust creates capture pipes before setting FD_CLOEXEC, so a simultaneous Nextest spawn can inherit another test's capture writer. The event-triggered wrapper recorded two cases: func::549 held func::555's writer beyond the 200 ms leak deadline, while func::633 inherited both ends of func::631's pipe. Both observations match rust-lang/rust#95584.

nextest-rs/nextest#3553 fixes the upstream spawn boundary. Its lock covers only macOS pipe setup through successful spawn, leaving child execution concurrent. The public high-concurrency reproducer leaked in three of five runs on Nextest 0.9.143 but passed 5,000 of 5,000 tests with the patch. Keep this issue open until Peryx can use a Nextest release containing the fix.

[gaborbernat]

An isolated #1548 full-package validation hit the same race after all assertions in
app::bootstrap::tests::test_bootstrap_administrator_contextualizes_filesystem_failures passed.

At Nextest's one-second EOF deadline, PID 20890 still held fd 49 for pipe inode 3463976253608257835. Its parent was
PID 98742. The retaining process ran the concurrent test
app::bootstrap::tests::test_bootstrap_administrator_accepts_password_length_boundaries::case_1_minimum.

A corrected diagnostic recorded fd 1, fd 2, and every inherited descriptor for all 905 tests. It passed without another
inherited descriptor at the deadline. The evidence matches the intermittent macOS spawn-boundary race already identified
here, not a lifecycle defect in either bootstrap test or the #1548 shutdown path.

[gaborbernat]

The workspace archive failure in run 29fb4861-14bf-48c9-aaba-a98842a6a5a6 is another instance of the
capture-inheritance race. tests::api::shared_tests::test_repository_state_filter_has_a_closed_enum exited with code
zero, then an unrelated process retained its capture pipe past the one-second deadline.

A native pre-exec wrapper recorded every pipe descriptor with proc_pidinfo and PROC_PIDFDPIPEINFO. The wrapper takes
its snapshot before exec starts the Peryx test binary. In archived run 5b867fcc-b974-4ea8-9b10-9acf5a4edec8, PID
17089 ran tests::cli::revocation_tests::test_parse_revocation_put_with_stdin_password with stderr pipe endpoint
107849119869742216. PID 17100 ran
tests::identity_availability_tests::test_permission_revoked_before_use_denies_the_action::case_1_none while holding
the same endpoint on inherited fd 31 and its peer 13978265745345159187 on inherited fd 28. PID 17089 also inherited
both ends of another capture pipe on fd 29 and fd 30. All four inherited descriptors had FD_CLOEXEC unset.

The target OpenAPI test passed by itself in 0.021 seconds and had only fd 1 and fd 2. Archive reuse is not causal.
Current-main source run ff3c6762-1885-4167-ba94-b41a1f66a41c reproduced the inheritance across 847 tests: PID 34612
held fd 34 for pipe endpoint 6033857739246077281 before exec, while PID 34613 used that endpoint for stderr. No Peryx
process or fixture created any of these descriptors because the wrapper observed them before the Peryx binary started.

The evidence matches the non-atomic pipe() and FD_CLOEXEC window documented in
rust-lang/rust#95584.
nextest-rs/nextest#3553 owns the spawn-boundary fix. This issue
remains blocked until a Nextest release contains that fix.

[gaborbernat]

The #1796 validation run reported LEAK for peryx-events::metrics::tests::test_drill_returns_repository_and_global_summaries. The descriptor evidence ties it to the spawn-boundary cause tracked here.

A native target runner recorded descriptors before each Peryx test binary executed. One unchanged peryx-events run with 64 test threads found PID 53562 holding non-CLOEXEC pipe fd 51 and PID 53724 holding non-CLOEXEC fds 139 and 141, both endpoints of one pipe. Cargo-nextest PID 52750 spawned both wrappers. Their stdout and stderr used different pipe handles. The metrics test starts no subprocess, and the target runner found these descriptors before Peryx code ran.

nextest-rs/nextest#3553 fixes the spawn boundary and remains open; the latest cargo-nextest release is 0.9.143. Peryx cannot prevent descriptor creation after its test binary starts. A descriptor-closing wrapper or serialized execution would mask the Nextest defect, while shutting down this test's in-process thread would address an unrelated resource.

[gaborbernat]

Nextest run a3096c33-f1de-45d2-8a4a-844e9e99b531 started five #1438 tests across three binaries and reported another
capture-inheritance failure after
tests::search::indexing::test_search_text_budget_preserves_each_field_class::case_1_summary_short returned code zero
in 0.775 seconds.

The selected search paths contain no operating-system process spawn. Their fixtures use temporary directories, redb,
filesystem BlobStorage, and Wiremock's in-process Tokio server. The kernel closes a process's descriptors when it
exits, so a separate process had to retain the capture writer. Nextest creates concurrent test processes; the selected
Peryx code creates none.

Diagnostic run a5aed7ca-96f5-457e-964d-3a9fe8c7403a executed the current #1438 source through the native pre-exec
descriptor recorder, which logged five invocations before Peryx started. Each process held fd 1 and fd 2, with no
inherited pipe at fd 3 or above. All five tests passed. No descendant survived. This sample rules out a persistent Peryx
descriptor owner, but it cannot name the transient owner from the uninstrumented failure.

The failure has the same process boundary as the captures above, including the earlier current-main run that found
another Nextest child holding a capture pipe before Peryx executed. Cargo-nextest 0.9.143 remains the latest release,
and nextest-rs/nextest#3553 remains open. The #1438 search change
should not add shutdown code for a process it does not start; this issue remains blocked on a released upstream fix.

[gaborbernat]

Two more current-main validations hit the capture-inheritance marker. Nextest run dc31d560-60f7-4d4e-b117-558fa98b173b
passed all 27 OCI system tests but reported LEAK for
peryx-oci-system-tests::ui cases::test_ui_oci_layer_lists_and_previews_members. Coverage run
33c6e7bd-0088-437b-8886-4ce5f187b5d0 passed all 3,279 tests but reported LEAK for
peryx-driver discovery::tests::test_base_url_from_request_rejects_invalid_forwarded_origin::case_1_no_headers. Neither
test starts an operating-system process.

Controlled OCI run 6372a2bf-4818-4252-b53b-dcb06f8bfc27 sampled every test process and its file descriptors. All 25
tests passed without LEAK; PID 26087 ran the affected layer-preview test, and no other sampled process held its stdout
or stderr endpoint. The sampler observed the expected descriptors inherited by each prefetch test's peryx child and
matched their shared endpoints. The sample rules out a persistent owner but does not identify the process from the
uninstrumented OCI failure.

#1629 already records native pre-exec captures from Peryx runs. They prove that sibling Nextest children receive
non-CLOEXEC capture endpoints before Peryx code starts.
nextest-rs/nextest#3553 applies its spawn boundary to every Unix
target outside
Rust's pipe2(O_CLOEXEC) list, rather
than to macOS alone. The two new failures remain blocked on a Nextest release containing that upstream fix; neither
Peryx test owns a shutdown path to repair.

[gaborbernat]

An exact current-main validation for #1145 passed all 110 selected tests but reported LEAK for
app::retention::tests::test_retention_public_entrypoint_reports_a_missing_store after the test returned code zero.
The selected code path does not start an operating-system process.

A dynamic interposer recorded process creation and capture-pipe identities for the exact test and the full 110-test
slice. The exact binary emitted no posix_spawn, fork, or execve event. Nextest run
90ae73f0-0f79-48e7-b426-3dda28afa94e passed all 110 tests; no simultaneous process held the affected test's stdout
or stderr endpoint.

Run 348adb72-c09e-41ce-8ad0-f04168799b98 also inspected every FIFO descriptor from fd 3 through fd 4095 before each
test binary executed. PID 64432 ran the affected test with only its fd 1 and fd 2 capture endpoints, inherited no extra
pipe, started no child, and returned code zero. All 110 tests passed. These samples do not recover the transient holder
from the uninstrumented failure. They rule out a persistent owner in the retention path and match this issue's
pre-exec evidence of sibling-process capture inheritance. Peryx made no timeout, serialization, wrapper, or lifecycle
change.

[gaborbernat]

Two more capture-inheritance victims, forwarded from #1869. I closed that issue as a duplicate of this one, matching
the disposition of #1826.

• peryx app::bootstrap::tests::test_bootstrap_administrator_contextualizes_filesystem_failures,
  LEAK-FAIL [1.129s] at 60698d6f.
• peryx-ecosystem-oci tests::virtual_tests::test_virtual_tags_union_hosted_and_upstream, LEAK [0.804s] on run
  af35b0be-727e-41d9-90e1-b1e691581c7c, aarch64-apple-darwin, 898 tests passed, exit code 0.

Both match the shape recorded here. The bootstrap case sits beside an inventory entry already listed above: it lives in
crates/peryx/tests/unit/tests/app/bootstrap_tests.rs, next to retention_tests.rs, whose
test_retention_public_entrypoint_reports_a_missing_store failed the same way. I validated it at current main
e4262e97, where the test and crates/peryx/src/app/bootstrap.rs match 60698d6f byte for byte. Both filesystem
error paths return before tokio::runtime::Builder::new_current_thread(). Neither the module nor the test file
contains Command, process::, spawn, or thread::. Nextest run d4c1a627-0cf2-4076-9e48-1d667ba4c3de then ran
the 12 selected test_bootstrap_administrator tests under --profile ci at that commit and reported 12 passed in
5.647s, none leaky. The OCI case at crates/peryx-ecosystem-oci/tests/unit/tests/virtual_tests.rs:503 builds an
in-process router and a wiremock server and starts no process.

#1869 also carried a syscall trace naming the writer for the bootstrap failure. Nextest 0.9.143 created the victim's
stderr pipe at 179166288138375 with writer handle 9056323228727010007 lacking CLOEXEC and applied FIOCLEX at
179166290053666. Concurrent test processes copied the writer as inherited fd 59 inside that window, and four holders
exited at 179168798196166, 179168804392541, 179168860164500, and 179169595483208, past the one-second deadline.

nextest-rs/nextest#3553 remains open and 0.9.143 remains the latest
cargo-nextest release.

[gaborbernat]

Another victim of the nextest/rustc pipe()-then-set_cloexec() race, seen while gating #2014: peryx-ecosystem-pypi tests::http::shadow::test_a_protected_name_denies_the_route_the_replay_reports reported LEAK on a spawn-free test (3566 passed, 1 leaky). No action taken.

[gaborbernat]

Another victim, spawn-free, from the #1299 lane: peryx-ecosystem-pypi tests::conformance_tests::test_upload_conformance_accepts_md5_only_digest reported LEAK on one cargo llvm-cov nextest -p peryx-archive -p peryx-ecosystem-pypi --all-features run and passed clean on the two runs either side of it, same code, same machine. The test spawns no process or thread (no Command, process::, thread::spawn in conformance_tests.rs), so this matches the pipe()-then-set_cloexec() race already tracked here rather than anything in that test. Recording only; no change made.