test: prove subprocess and production coverage (Fixes #534) (#562)

Make coverage evidence demonstrate real server execution and expose
production gaps without changing existing whole-workspace regression
gates.

- Compare exact idle/info child profiles and require positive execution
in the real handler, transport dispatch, and response writer after
graceful shutdown.
- Add production/test and changed-line diagnostics with conservative
accounting for LLVM summary entries without unique source lines.
- Require profile proof and reporting in Linux/Windows PR and baseline
jobs; add native macOS coverage measured against the exact base on the
same runner.
- Document classification, branch-instrumentation limits, and artifact
semantics.

Validation: 76 Python tests, 19 Windows native cases, workspace
formatting/Clippy, and independent review pass. Actual Windows LLVM
profiles prove all three execution witnesses increase from 0 to 1.
Hosted macOS validation and quality inspection remain pending; no
signing/release services run.

Fixes #534

---------

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Author: karthiknadig

.github/skills/rust-coding-skill/SKILL.md

@@ -102,3 +102,5 @@ Do not execute freshly written scripts as concurrent Unix subprocess fixtures: s
 For real-pipe EOF/EPIPE tests, create the pipe inside an isolated test subprocess when other test threads spawn children. Unix `CLOEXEC` closes descriptors at exec, not fork: a concurrent child can temporarily retain a reader, allowing the only write to succeed before the final reader disappears. A readiness handshake alone does not prevent this race. Keep the operation's measured deadline separate from setup, and make an outer fixture deadline cover readiness, waits both before and after forced termination, reader joins, and fallback `Drop` cleanup.
 
 Use a per-worktree Cargo target directory when validating stacked changes so native fixtures cannot execute another worktree's stale binary. On WSL, run timing-sensitive Linux binaries from the native Linux filesystem rather than a Windows mount, where page faults can stall in filesystem RPC. When launching instrumented PET with `env_clear()`, retain `LLVM_PROFILE_FILE` exactly so child coverage reaches the collector instead of an uncollected default profile.
+
+LLVM LCOV summaries are not necessarily counts of unique `DA` source entries: a native export can have `LF=191`, `LH=173`, 181 mapped lines, and 177 positive mapped lines. Do not reject valid exports using summary/source equality or silently drop unmapped entries; preserve the raw exact-base gate and follow the conservative supplemental accounting documented in [Quality snapshots](../../../docs/QUALITY_SNAPSHOTS.md#production-focused-and-subprocess-evidence). Prove subprocess collection with exact-PID profiles and isolated idle/request counter differences, not a whole-workspace percentage increase.

.github/workflows/coverage-baseline.yml

@@ -27,6 +27,9 @@ jobs:
     steps:
       - name: Checkout
         uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
+        with:
+          fetch-depth: 0
+          persist-credentials: false
 
       - name: Set Python to PATH
         uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v7.0.0
@@ -181,6 +184,21 @@ jobs:
         env:
           RUST_BACKTRACE: 1
           RUST_LOG: trace
+          PET_SUBPROCESS_COVERAGE_PROOF: ${{ github.workspace }}/subprocess-coverage.json
+        shell: bash
+
+      - name: Verify Isolated Server Coverage
+        if: always()
+        run: python scripts/coverage_detail.py proof --manifest subprocess-coverage.json --output subprocess-coverage
+        shell: bash
+
+      - name: Report Production and Changed Coverage
+        if: always()
+        run: >-
+          python scripts/coverage_detail.py report
+          --lcov lcov.info
+          --base "HEAD"
+          --output production-coverage
         shell: bash
 
       - name: Validate Coverage Baseline
@@ -194,8 +212,14 @@ jobs:
         shell: bash
 
       - name: Upload Coverage Artifact
+        if: always()
         uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
         with:
           name: coverage-baseline-${{ matrix.os }}
-          path: lcov.info
+          path: |
+            lcov.info
+            coverage-baseline-report.md
+            production-coverage/
+            subprocess-coverage/
+            subprocess-coverage.json
           retention-days: 90

.github/workflows/coverage-macos.yml

@@ -0,0 +1,93 @@
+name: Native macOS Coverage
+
+on:
+  pull_request:
+    branches: [main, 'release*', 'release/*', 'release-*']
+  push:
+    branches: [main, 'release*', 'release/*', 'release-*']
+  workflow_dispatch:
+
+permissions:
+  contents: read
+
+jobs:
+  coverage:
+    runs-on: macos-14
+    timeout-minutes: 35
+    steps:
+      - name: Checkout
+        uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
+        with:
+          fetch-depth: 0
+          persist-credentials: false
+
+      - name: Set Python to PATH
+        uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v7.0.0
+        with:
+          python-version: '3.12'
+
+      - name: Rust Tool Chain setup
+        uses: dtolnay/rust-toolchain@stable
+        with:
+          toolchain: stable
+          components: llvm-tools-preview
+
+      - name: Install cargo-llvm-cov
+        uses: taiki-e/install-action@cargo-llvm-cov
+
+      - name: Validate Coverage Reporting
+        run: python -m unittest discover -s scripts/tests -p 'test_*.py' -v
+
+      - name: Collect Native macOS Coverage
+        run: cargo llvm-cov --workspace --lcov --output-path lcov.info -- --test-threads=1
+        env:
+          PET_SUBPROCESS_COVERAGE_PROOF: ${{ github.workspace }}/subprocess-coverage.json
+
+      - name: Verify Isolated Server Coverage
+        if: always()
+        run: python scripts/coverage_detail.py proof --manifest subprocess-coverage.json --output subprocess-coverage
+
+      - name: Report Production and Changed Coverage
+        if: always()
+        env:
+          BASE: ${{ github.event.pull_request.base.sha || github.sha }}
+        run: python scripts/coverage_detail.py report --lcov lcov.info --base "$BASE" --output production-coverage
+
+      - name: Measure Exact PR Base on the Same Runner
+        if: github.event_name == 'pull_request'
+        env:
+          BASE: ${{ github.event.pull_request.base.sha }}
+          CARGO_TARGET_DIR: ${{ runner.temp }}/coverage-base-target
+          PET_SUBPROCESS_COVERAGE_PROOF: ${{ runner.temp }}/base-subprocess-coverage.json
+        run: |
+          git worktree add --detach "$RUNNER_TEMP/coverage-base" "$BASE"
+          cd "$RUNNER_TEMP/coverage-base"
+          cargo llvm-cov --workspace --lcov --output-path "$GITHUB_WORKSPACE/baseline-lcov.info" -- --test-threads=1
+
+      - name: Compare Exact Base Coverage
+        if: always() && github.event_name == 'pull_request'
+        run: >-
+          python scripts/quality_snapshot.py coverage
+          --current lcov.info --baseline baseline-lcov.info --platform macOS
+          --report coverage-report.md --summary "$GITHUB_STEP_SUMMARY"
+
+      - name: Validate Main Coverage
+        if: github.event_name != 'pull_request'
+        run: >-
+          python scripts/quality_snapshot.py coverage
+          --current lcov.info --baseline lcov.info --platform macOS
+          --report coverage-report.md --summary "$GITHUB_STEP_SUMMARY"
+
+      - name: Upload Native macOS Coverage
+        if: always()
+        uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
+        with:
+          name: coverage-macos-native
+          path: |
+            lcov.info
+            baseline-lcov.info
+            coverage-report.md
+            production-coverage/
+            subprocess-coverage/
+            subprocess-coverage.json
+          retention-days: 30

.github/workflows/coverage.yml

@@ -32,6 +32,9 @@ jobs:
     steps:
       - name: Checkout
         uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
+        with:
+          fetch-depth: 0
+          persist-credentials: false
 
       - name: Post Coverage Started Comment
         if: github.event.pull_request.head.repo.full_name == github.repository
@@ -200,6 +203,21 @@ jobs:
         env:
           RUST_BACKTRACE: 1
           RUST_LOG: trace
+          PET_SUBPROCESS_COVERAGE_PROOF: ${{ github.workspace }}/subprocess-coverage.json
+        shell: bash
+
+      - name: Verify Isolated Server Coverage
+        if: always()
+        run: python scripts/coverage_detail.py proof --manifest subprocess-coverage.json --output subprocess-coverage
+        shell: bash
+
+      - name: Report Production and Changed Coverage
+        if: always()
+        run: >-
+          python scripts/coverage_detail.py report
+          --lcov lcov.info
+          --base "${{ github.event.pull_request.base.sha }}"
+          --output production-coverage
         shell: bash
 
       - name: Wait for Exact PR Base Coverage
@@ -248,6 +266,9 @@ jobs:
           path: |
             lcov.info
             coverage-report.md
+            production-coverage/
+            subprocess-coverage/
+            subprocess-coverage.json
           if-no-files-found: ignore
 
       - name: Post Coverage Comment

crates/pet/tests/jsonrpc_server_test.rs

@@ -38,13 +38,20 @@ struct RawRpcClient {
 
 impl RawRpcClient {
     fn spawn() -> Self {
+        Self::spawn_with_profile(None)
+    }
+
+    fn spawn_with_profile(profile: Option<&Path>) -> Self {
         let mut command = Command::new(env!("CARGO_BIN_EXE_pet"));
         command
             .arg("server")
             .stdin(Stdio::piped())
             .stdout(Stdio::piped())
             .stderr(Stdio::inherit());
         jsonrpc_client::configure_isolated_pet_environment(&mut command);
+        if let Some(profile) = profile {
+            command.env("LLVM_PROFILE_FILE", profile);
+        }
         let mut child = command.spawn().expect("raw fixture must spawn PET");
         let stdout = child.stdout.take().expect("PET stdout must be piped");
         let (sender, responses) = mpsc::channel();
@@ -819,6 +826,89 @@ fn invalid_and_oversize_framing_terminates_with_bounded_diagnostics() {
     }
 }
 
+#[test]
+fn normal_shutdown_records_pid_unique_server_profiles() {
+    let Some(proof_path) = std::env::var_os("PET_SUBPROCESS_COVERAGE_PROOF") else {
+        return;
+    };
+    let inherited = PathBuf::from(
+        std::env::var_os("LLVM_PROFILE_FILE")
+            .expect("coverage proof requires cargo-llvm-cov instrumentation"),
+    );
+    let directory = inherited.parent().expect("profile must have a directory");
+    assert!(
+        directory.is_absolute(),
+        "profile directory must be absolute"
+    );
+    assert!(
+        inherited
+            .file_name()
+            .unwrap()
+            .to_string_lossy()
+            .contains("%p"),
+        "cargo-llvm-cov must use PID-unique profiles"
+    );
+    let mut profiles = serde_json::Map::new();
+    for (name, request) in [("idle", false), ("info", true)] {
+        let prefix = format!("pet-proof-{}-{name}-", std::process::id());
+        let pattern = directory.join(format!("{prefix}%p-%m.profraw"));
+        let mut client = RawRpcClient::spawn_with_profile(Some(&pattern));
+        let prefix = format!("{prefix}{}-", client.child.id());
+        assert!(
+            fs::read_dir(directory).unwrap().all(|entry| !entry
+                .unwrap()
+                .file_name()
+                .to_string_lossy()
+                .starts_with(&prefix)),
+            "child profile must not predate this process exit"
+        );
+        if request {
+            client.send(json!({"jsonrpc": "2.0", "id": "profile-proof", "method": "info"}));
+            let reply = client.receive();
+            assert_eq!(reply["id"], "profile-proof");
+            assert_eq!(reply["result"]["petVersion"], env!("CARGO_PKG_VERSION"));
+        }
+        client.child.stdin.take();
+        let status = jsonrpc_client::wait_for_exit(&mut client.child, Duration::from_secs(4))
+            .expect("profile proof requires graceful exit, not kill fallback");
+        assert!(
+            status.success(),
+            "profile probe exited unsuccessfully: {status}"
+        );
+        jsonrpc_client::join_reader(client.reader.take().unwrap(), Duration::from_secs(4)).unwrap();
+        let raw: Vec<PathBuf> = fs::read_dir(directory)
+            .unwrap()
+            .map(|entry| entry.unwrap().path())
+            .filter(|path| {
+                path.file_name()
+                    .unwrap()
+                    .to_string_lossy()
+                    .starts_with(&prefix)
+            })
+            .collect();
+        assert!(
+            !raw.is_empty(),
+            "normal server exit did not flush its own profile"
+        );
+        for path in &raw {
+            assert!(
+                fs::metadata(path).unwrap().len() > 0,
+                "empty server profile"
+            );
+        }
+        profiles.insert(name.into(), json!(raw));
+    }
+    fs::write(
+        proof_path,
+        serde_json::to_vec_pretty(&json!({
+            "binary": env!("CARGO_BIN_EXE_pet"),
+            "profiles": profiles,
+        }))
+        .unwrap(),
+    )
+    .expect("write subprocess coverage evidence");
+}
+
 #[test]
 fn stdin_eof_after_exchange_exits_cleanly_within_one_second() {
     let client = PetJsonRpcClient::spawn().unwrap();

docs/QUALITY_SNAPSHOTS.md

@@ -113,6 +113,61 @@ for the new client clocks.
 
 Linux and Windows line and function coverage are compared with the exact base commit. A decrease greater than 0.01 percentage points blocks the pull request. Coverage artifacts and comments remain available for inspection even when the comparison fails.
 
+### Production-focused and subprocess evidence
+
+The raw workspace percentages still include inline tests and retain the same exact-base
+0.01 percentage-point line/function gate. Supplemental `production-coverage/report.md` and
+schema-1 `details.json` separate executable production/test lines, list uncovered production
+lines, and intersect added/modified Rust lines with executable production lines. Changed files
+without instrumentation are listed explicitly, never assumed covered. These diagnostics do not
+introduce a fabricated baseline, change the raw denominator, or replace regression protection.
+
+Classification excludes integration-test/benchmark directories and Rust items explicitly marked
+`#[cfg(test)]` or `#[test]`, including inline modules and test-only helper functions. Nested
+`all`/`any` predicates are also excluded when they require `test`: `all(test, unix)` is test-only,
+but `any(test, unix)` is not. It masks
+strings, raw/byte strings, characters, and nested comments before matching item boundaries.
+Helpers outside those boundaries and unsupported conditional predicates remain conservatively in the
+production category; this is a source-focused diagnostic, not full Rust conditional-compilation
+analysis. Invalid/missing LCOV, missing source, inconsistent hit summaries, and source-line
+mismatches fail the reporting step. LLVM summaries can include more entries in `LF`/`LH`
+than the unique `DA` source lines (observed in real Windows exports). That deficit is reported
+per file (including unmatched summary hits) and conservatively retained as uncovered production
+in mixed source files, or uncovered tests in integration-test/benchmark files. It is never dropped
+from the denominator or silently assigned coverage.
+Changed lines without `DA` records are listed separately in JSON, including non-executable syntax;
+they are not silently considered covered.
+Native macOS also demonstrates `LH` below the number of positive unique `DA` entries. Reports
+retain this deficit and deduct `max(positive DA + unmapped LF - LH, 0)` from each covered
+production/test/changed subtotal (clamped at zero). This accounts for hits that could belong to
+unmapped entries instead of a mapped subset. These subtotals are lower bounds; the report does
+not pretend to locate the discrepancy on a particular source line. Raw LCOV and the exact-base gate remain intact.
+
+Every coverage job opts into `normal_shutdown_records_pid_unique_server_profiles` through
+`PET_SUBPROCESS_COVERAGE_PROOF`. The test requires cargo-llvm-cov's absolute, PID-unique output
+pattern, launches idle and known-`info` PET subprocesses, closes stdin, and requires successful
+bounded exit and nonempty profiles for those exact child PIDs. The raw profiles stay in the normal
+cargo-llvm-cov collection directory and are included in the workspace report. The verifier also
+merges each child's profiles separately with the matching Rust LLVM tools and proves zero idle
+versus positive `info` execution at the real handler, transport dispatch, and response writer.
+The uploaded `subprocess-coverage/proof.json` and isolated LCOV exports retain that evidence;
+a killed child, missing profile, or absent execution witness fails rather than appearing covered.
+
+Native ARM64 macOS coverage runs workspace default-feature and native process/transport tests,
+including Darwin-specific process ownership paths. It intentionally does not compare this workload
+with Linux/Windows's installed-manager `ci` workload. For each macOS PR, the exact base revision is
+built and measured separately on the same runner with the same compiler and feature selection;
+the unchanged line/function comparator gates those comparable artifacts. This works on the first
+PR without silently accepting an absent macOS baseline. Main/manual runs publish the native
+measurement and proof for inspection. Existing functional macOS installed-manager jobs remain.
+
+Stable Rust line instrumentation does not provide condition/branch outcomes. Reports show LCOV
+`BRDA` totals when supplied, otherwise explicitly report branch data as unavailable (not 100%).
+Native malformed-frame/envelope, EOF, broken-output, saturation, and descendant tests provide
+behavioral failure-path evidence, but line coverage cannot prove both sides of every condition.
+Nightly `cargo llvm-cov --branch` can be used as a separate experiment; its unstable toolchain
+and differing denominator are not substituted into the stable cross-platform gate.
+
 ## Running locally
 
 The comparator requires Python 3.10 or newer.