test: investigate intermittent nextest resource leak annotations

[jscott3201]

Observation

During macOS nextest validation of 2571783e2557f9d98aeec97120b1caa6989cb288, the feature-enabled workspace run succeeded with 5,624 passed and two skipped, but annotated one passing test:

LEAK [0.209s] bacnet-server server::segmentation_tests::control_limits::segmented_complex_ack_rejects_new_sender_when_active_sender_limit_reached

A single focused rerun passed in 0.019s without LEAK. Cause and baseline attribution are unverified. This is a test-process/resource or output-handle lifetime observation, not proof of a production memory leak or an introduced regression. Source was unchanged between runs.

Investigation and acceptance

Reproduce under representative concurrent workspace load and compare with the pre-change baseline. Determine which process/output handle outlives test completion; inspect fixture teardown and runtime task ownership. Add a bounded deterministic regression and repair the owning lifetime if a defect is confirmed. Do not suppress the warning, add retries/skips, or extend leak timeouts to obtain a pass without identifying the cause. If investigation cannot reproduce it, retain the evidence and clearly label the result.

Focused command:

cargo nextest run -p bacnet-server --locked --features bacnet-types/serde,bacnet-transport/ipv6,bacnet-transport/sc-tls -E 'test(=server::segmentation_tests::control_limits::segmented_complex_ack_rejects_new_sender_when_active_sender_limit_reached)' --no-tests=fail

Local session evidence: /private/tmp/remove-rollback-workspace.log and /private/tmp/remove-rollback-leaky-rerun.log. Recorded as a follow-up to the sixth PR; no failure was waived and no seventh PR is part of this batch.

Second observation — 23 September 2026

During #752 validation in a worktree based on 574be594baa784acd57aad2bfb4f038a25399e1c, the macOS serde/IPv6/SC-TLS workspace run exited successfully: 5,660 passed (one leaky), two skipped. The benchmark test source was unchanged from that base. A different passing test was annotated:

LEAK [0.609s] bacnet-benchmarks::sc_binary preflight::non_sc_bip_starts_without_credentials_and_panic_reaps_child

No symptom-only rerun, suppression or timeout change was made. Local evidence is /private/tmp/life-safety-contract-workspace.log, annotation at line 6240 and final summary at line 11869.

Independent read-only inspection found that benchmarks/tests/sc_binary/preflight.rs moves the child guard into the deliberately panicking caught future, then verifies its UDP port can be rebound. support.rs sends child stdout/stderr to regular files, with no pipe-reader task; Process::drop calls kill and wait synchronously. No grandchild spawn was found in the inspected device startup path. Drop ignores kill/wait results, so rebinding proves socket closure without independently proving successful reaping. This is a limitation of the evidence, not proof that cleanup failed. The earlier segmentation rejection path creates no child or dispatch task. A common cause is unproven.

Keep both observations in this investigation until a controlled reproduction identifies remaining process IDs, wait results or inherited capture descriptors at test-process exit. Split a specific fixture/production repair only when that ownership evidence supports it. Do not infer a heap leak or blame the LifeSafety contract change from this annotation.

Third observation — 23 September 2026, target recipient work

During the feature-enabled macOS workspace validation of target recipient work based on ba1ad436d2555a7f7e38e703a100562e4c641f59, the corrected repository-standard selection (--workspace --exclude rusty-bacnet) exited successfully: 5,681 passed, including one leaky annotation, and two skipped:

LEAK [0.251s] bacnet-benchmarks::sc_mtls node_reconnect::node_tls_client_builder_preflight_keeps_existing_error_precedence

Evidence: /private/tmp/recipient-workspace-nextest-corrected.log, annotation at line 6074 and summary at line 11752. No warning suppression, timeout extension or symptom-only rerun was performed.

The reached test body is benchmarks/tests/sc_mtls/node_reconnect.rs:280-307: it generates certificates with the in-process rcgen helper, then checks three invalid builder options. ScClientBuilder::build rejects those options before TlsWebSocket::connect or client startup (crates/bacnet-client/src/client/mod.rs:758-777). No explicit application child process, Hub, proxy or endpoint task is created by this test path. The benchmark test, certificate helper and client builder source are unchanged from the stated base, as verified by Git diff. This does not prove what native/runtime or runner-owned resource caused the annotation, nor attribute it to the recipient change.

The observations now span a segmentation preflight, a child-process cleanup fixture, and an SC builder validation test. Continue controlled investigation of surviving process/capture descriptors and runner behavior under concurrency before assigning a shared cause. There is still no proven production heap leak or reproducible owner defect to repair.

Fourth observation — 23 September 2026, conflict-aware Hub admission

During the first focused macOS admission run in the worktree based on fa88b1ddb963271fde411c9d09ddd295c768c425, nextest exited successfully with 21 passed (one leaky) and 930 skipped:

LEAK [0.251s] bacnet-transport sc::source_admission_tests::hub_npdu_missing_source_unicast_returns_connection_local_nak

Command: cargo nextest run -p bacnet-transport --features sc-tls -E 'test(admission_tests)' --no-tests=fail --locked. Run ID f7e24515-cea4-4efa-870e-833eb12ab5af; default profile, no test-thread override or NEXTEST_TEST_THREADS, host with 10 logical CPUs, one authorized Cargo owner. The source-admission test matches the filter name but is separate from the new Hub registration-policy regressions.

The named test and its sc/data_attribute_tests.rs helper are unchanged from the stated base. The visible fixture creates a LoopbackWebSocket pair, joins its connection-accept task, validates the Result, and awaits transport.stop(). This observation does not identify a surviving process or descriptor, or establish a common cause with the previous annotations. Original log: /private/tmp/sc-conflict-admission-focused.log (annotation line 2573, summary line 2577); run metadata: /private/tmp/sc-conflict-admission-leak-observation.json. No symptom-only retry, suppression, skip or leak-timeout change was made. Keep controlled process/capture-descriptor investigation open.

Diagnostic preparation — 24 September 2026 (no new reproduction)

Installed cargo-nextest is 0.9.143, commit 60fa45f638ffc3f35e74afa65737f45fcd32db2a, on aarch64-apple-darwin. There is no repository .config/nextest.toml at this checkpoint. Its pinned embedded configuration uses a 200 ms leak timeout; the current general leaky-tests web page still describes 100 ms, so use the actual installed revision and effective arguments/configuration for the experiment.

The pinned detector waits for captured stdout/stderr completion after test-process exit; a LEAK label alone does not identify a surviving process or heap leak. Preserve normal output capture and representative concurrency when collecting PID/PGID and descriptor ownership evidence. Compare the same bounded selection/features on the recorded baseline and current code; separate observer overhead and no-symptom runs from causal proof.

Do not treat --tracer or --no-capture as an equivalent leak reproduction: installed help shows tracer mode disables capture/timeouts, and the pinned interceptor skips leak detection. No-capture also serializes the runner. An external observer (or available USDT lifecycle probes) should retain the original detection path; verify availability before choosing it. Traced single-test runs may answer a different ownership question but cannot establish that the concurrent captured-run symptom disappeared.

Only version/help/source inspection was performed for this preparation. No stress run, configuration/timeout change, suppression, runner upgrade or competing Cargo workload was started. The four original observations and unresolved cause remain intact.

Fifth observation — 24 September 2026, COV identity qualification

On the frozen COV identity tree now committed as 5617bc1edfb36d59f878e810babeb5ad7554bb6c (base e713b241ff6f4de47425cde3315c852c88ff7212), the macOS integration selection passed 10 tests, with one leaky annotation and 83 tests excluded:

LEAK [0.216s] bacnet-integration-tests::server error_cov::read_nonexistent_object_returns_unknown_object_error

Command: cargo nextest run --locked -p bacnet-integration-tests -E 'test(cov)' --no-tests=fail --no-fail-fast. Default concurrency, no competing build/test, authorized loopback execution. The module name error_cov includes this ReadProperty error-path test in the COV selection. The preserved log is /private/tmp/rb-cov-identity-integration.log (annotation line 5504, summary line 5509, log timestamp 2026-09-24 09:11:12 UTC); run summary was 10 passed (1 leaky) in 0.518 seconds.

The test and its tests/server.rs helpers are unchanged from the stated base. The visible test creates a real loopback B/IP server/client, reads a nonexistent Analog Input property, checks OBJECT/UNKNOWN_OBJECT, then awaits both client and server stop. It creates no COV subscription or explicit child process. Server implementation inputs changed in this PR, so unchanged fixture source does not establish baseline attribution. A later safe process inventory found no matching Cargo/native/test processes or target executable paths; it did not observe the capture descriptors at the moment of the annotation.

No symptom-only rerun, suppression, retry, skip, leak-timeout change or runner upgrade was performed. Preserve this observation with the four earlier ones. Cause, shared ownership and baseline reproducibility remain unverified; the next useful investigation is the controlled process/capture-descriptor comparison described above.

Sixth observation — 24 September 2026, monitored-property COV qualification

While qualifying the typed property-COV change based on 678ad3b5866c41a6cef703f89dd6c165fca42ae6, the first macOS affected selection ran 1,645 tests: 1,625 passed (one leaky), 20 failed, 714 excluded. Nineteen failures were explicit sandbox loopback EPERM errors; the remaining failure was a Life Safety fixture expecting unchanged selected state to notify. Preserve this initial result separately from subsequent qualification.

LEAK [0.212s] bacnet-client client::tests::confirmed_request_does_not_wrap_discovered_max_apdu_length

Command: cargo nextest run --locked -p bacnet-server -p bacnet-client -p bacnet-services -E 'package(bacnet-server) | test(cov)' --no-tests=fail --no-fail-fast. Default nextest concurrency, sole Cargo owner, first command under the sandbox. Log /private/tmp/rb-cov-property-samples-affected-first.log, annotation line 5762 and summary line 7587; start/completion 2026-09-24 09:57:11–09:58:12 UTC. Details are preserved in rb-cov-property-samples-leak.json.

The named test at crates/bacnet-client/src/client/tests.rs:307 is unchanged from the stated base. It uses an in-memory LoopbackTransport pair, supplies an oversized discovered APDU length, receives a SimpleACK from a spawned responder, awaits that responder, and awaits client stop; the responder also stops its remote network. The named test does not open an OS loopback socket or explicitly spawn a child process. This does not identify a retained capture descriptor or establish baseline causality.

An authorized broader run was necessary for the 19 environment failures and fixture repair. It was not a symptom-only retry to clear LEAK; subsequent absence of an annotation does not resolve this investigation. No timeout change, warning suppression, retry policy or runner upgrade was introduced. Keep all six observations and the controlled capture-preserving diagnosis described above; cause and any shared owner remain unverified.

Seventh observation — 26 September 2026, endpoint WriteProperty qualification

The frozen tree committed as f8471da4276585504d813e0be3ceffe684f58932 (base ca3f9fe48fcaab164d6fa0b8b1b2a0e150b7f9a3, PR #864) passed the full local workspace selection: 6,029 tests passed, including one LEAK annotation, and 2 skipped.

LEAK [0.251s] bacnet-benchmarks::sc_mtls node_reconnect::node_tls_client_builder_preflight_keeps_existing_error_precedence

Command: cargo nextest run --workspace --exclude rusty-bacnet --locked --no-tests=fail --no-fail-fast. Original log /private/tmp/rb-endpoint-source-wp-workspace.log, annotation line 6125 and summary line 12151, elapsed 42.954s. The test in benchmarks/tests/sc_mtls/node_reconnect.rs:280 is unchanged from that base. This does not establish a baseline cause or identify a retained capture descriptor.

A subsequent isolated recheck passed 1 test (27 excluded) without the annotation in 0.047s; it is recorded as a no-symptom diagnostic, not a replacement for the original result or evidence that this issue is resolved. Original and recheck logs are retained under _spec/rusty-bacnet-release-plan/reviews/PR-864-evidence/round1-f8471da4/ with an artifact hash index. No timeout change, suppression, runner upgrade or test skip was introduced. A controlled, capture-preserving process/descriptor investigation remains outstanding.

Eighth observation — 27 September 2026, NORMAL B/IP Network Number qualification

While implementing #875 on branch codex/nonrouter-network-number from base 6ba0dfaf1f0ecaf4a5c29df2e220582356056443, a focused macOS run passed 14 tests (one leaky), with 3,833 tests excluded and exit status 0. This was a mutable work-in-progress run: no source hash manifest was captured, and the base commit must not be represented as its qualification head.

LEAK [0.224s] bacnet-endpoint session::network_number_tests::network_number_endpoint_answers_configured_local_what_is

Command: cargo nextest run -p bacnet-endpoint -p bacnet-endpoint-core -p bacnet-server -p bacnet-objects -p bacnet-transport --lib --locked --no-tests=fail --no-fail-fast -E 'test(network_number_) | test(original_broadcast_npdu_preserves)'. Installed nextest 0.9.143, default profile, no explicit environment overrides, authorized localhost socket access, one Cargo owner. Run ID 25b05a79-e540-465b-a123-b37325f9b645. Original log /private/tmp/rb-network-number-focused5.log; a copy and SHA-256 index are preserved under _spec/rusty-bacnet-release-plan/reviews/ISSUE-875-evidence/interim-diagnostics/.

This is an unexplained captured-output annotation, not a failed assertion or demonstrated BACnet resource leak. No captured process/descriptor observation identifies its cause or establishes a common cause with earlier observations. The subsequent expanded qualification passed 16 tests, including direct zero-live-Tokio-task, registration release, real UDP-port rebind, bare-Drop and independent InputClosed checks. Those are separate lifecycle evidence; absence of the annotation in that run does not resolve this issue. No capture suppression, timeout change, skip, retry-to-clear policy or runner upgrade was introduced. Preserve the original observation and continue the capture-preserving diagnosis above.

Ninth observation — 27 September 2026, Hub certificate-binding qualification

While implementing #800 on codex/sc-hub-certificate-bindings from base 650377f9023e4a0e4f2275518c842090f9b3ce29, a macOS workspace run exited0 with6,072 tests passed (one leaky), two policy skips,42.255s. This was mutable work in progress, before the final Option<VerifiedLeaf> cleanup; no immutable WIP source manifest existed. The base is not its qualification head.

LEAK [0.248s] bacnet-benchmarks::sc_mtls node_reconnect::node_tls_client_builder_preflight_keeps_existing_error_precedence

Command: cargo nextest run --workspace --exclude rusty-bacnet --locked --no-tests=fail --no-fail-fast, from /Users/justin/Development/rusty-bacnet, no explicit environment overrides, default profile and one Cargo owner. Run ID f7373c43-f8a9-4437-9c84-26bf6222ecac. Original log /private/tmp/rb-hub-bindings-workspace-final.log, completed2026-09-27T19:05:46.755474Z, SHA256b4e9d5c73e1f603ccc4c29e3abf2661422c5ff743d5dff45e632d3c08fb39184. Durable copy under _spec/rusty-bacnet-release-plan/reviews/ISSUE-800-evidence/interim-diagnostics/. Its filename does not imply the later source freeze.

Bounded source assessment: this fixture generates in-memory rcgen certificates, constructs node configuration, and exercises three invalid builder configurations that return before TlsWebSocket::connect at reconnect/identity/max-segments guards. It starts no Hub, does not enter the new binding path, and the certificate helper starts no child process. This narrows direct fixture-owned activity; it does not identify the capture annotation cause or establish a common cause with earlier observations. The first workspace run had no annotation. Later final-identity affected SC qualification passed984tests; neither absence of a repeated annotation nor source inspection resolves this observation. No suppression, capture change, timeout increase, skip or retry-to-clear policy was introduced. Keep the original observation and capture-preserving diagnosis open.

[jscott3201]

Cause: a race in cargo-nextest below 0.9.145 on macOS, not a fixture or production defect

Mechanism. On macOS, Rust's standard library creates a pipe with pipe() and sets close-on-exec in a separate call, because Apple platforms have no pipe2. nextest spawns tests from several threads, and its split capture uses Stdio::piped() (nextest-runner 0.122.1, src/test_command/imp.rs). A test process spawned in that window inherits a sibling test's capture pipe, including its write end.

nextest marks a test LEAK when its stdout or stderr doesn't reach EOF within leak-timeout, which defaults to 200 ms (default-config.toml, and detect_fd_leaks in src/runner/executor.rs). So the sibling is flagged even though it created nothing. Linux uses pipe2(O_CLOEXEC), which is why only macOS runs showed this. Upstream fixed it in nextest-rs/nextest#3553, released in cargo-nextest 0.9.145.

That explains all four observations: segmentation preflight, SC builder validation, child reaping, and SC Hub admission. They're unrelated tests, some of which spawn nothing, and their LEAK times were just over the timeout (0.209 s, 0.251 s, 0.609 s).

Evidence

Environment: macOS 27.2, Apple M5 with 10 CPUs, cargo-nextest 0.9.143 (the version this issue was recorded with).

I used a separate dependency-free crate. It had no child processes, no runtime and no project code: 300 trivial tests and 60 tests that sleep 1 s, run with -j 32.

• Inheritance observed directly. The long tests listed /dev/fd and found pipe descriptors above stderr that they never created, usually as adjacent pairs (both ends of one pipe). This happened in about 1 to 5% of long-test processes.

• LEAK reproduced without project code. Once short and long tests were interleaved by name, so short tests spawned while long ones were starting, trivial tests were annotated LEAK [0.209s] and LEAK [0.227s].

• Paired runner comparison. 20 runs each on the same crate and host:

  cargo-nextest	LEAK annotations
  0.9.143	7
  0.9.146 (scratch install)	0

The segmentation and SC fixtures named in this issue don't need changes. Rather than suppressing the warning or raising the timeout, the remedy is a runner of 0.9.145 or later. CI runs plain cargo test (.forgejo/workflows/ci.yml) and isn't affected.