Skip to content

test: name the run in e2e coverage tooling-error artifacts - #1070

Merged
castrojo merged 2 commits into
mainfrom
quality/e2e-report-run-identity
Oct 6, 2026
Merged

castrojo merged 2 commits into
mainfrom
quality/e2e-report-run-identity

Conversation

@hivecommons-hive

Copy link
Copy Markdown
Contributor

Test Improvement

main() in tests/tools/e2e-coverage-report.mjs built its failure artifact from
error.runId / error.runStatus. Only CoverageReportError carries those, and
only the three run-status/capture guards near the top of collectE2ECoverage
raise it. Every throw after the manifest has been read and validated — the
.tmp guard, the artifact-kind and run-id guards, every convertScript source-map
failure, and No src/** coverage was attributable — is a plain Error, so the
published header read:

E2E coverage: tooling-error (run null; status unknown)

even when manifest.json on disk said "runId": "run-1", "status": "passed".

.github/workflows/ci.yml cats coverage/e2e/report.txt straight into
$GITHUB_STEP_SUMMARY, and that line is the whole diagnostic. It could not
distinguish "the Playwright run failed" from "only the reporting step broke" —
two failures with opposite remedies.

This re-reads the manifest in the catch. readCoverageRun was already imported;
.catch(() => null) leaves an unreadable or absent manifest behaving exactly as
it does today.

Tests

Two cases appended to tests/e2e-coverage-report.test.mjs:

  • a tooling-error artifact still names the run the manifest records — seals a
    run passed, trips the No src/** path, and asserts the JSON carries
    runId: 'run-1' / runStatus: 'passed' and the text header reads
    E2E coverage: tooling-error (run run-1; status passed). Verified
    non-vacuous
    : reverting only the .mjs change fails this test and no other.
  • a tooling-error artifact reports no run when the manifest is unreadable —
    pins the unchanged null / null behaviour.

Scope note

Issue #1069's second completion box — a run sealed failed/cancelled reported
with that status rather than tooling-error — is not reachable and is
deliberately not implemented. collectE2ECoverage rejects every non-passed
run with a CoverageReportError before any plain-Error site runs, so the
manifest fallback can only ever observe passed, and the existing ternary on
error.runStatus already handles the sealed-failed case correctly (covered by
main labels failed runs as failed and exits nonzero). Adding the branch would
be dead code. I have said the same on the issue; nothing else is left for it to
track.

Verification

  • node --test tests/e2e-coverage-report.test.mjs — 37 passed, 0 failed.
  • npm run test:unit:coverage:check — exit 0 (--check-source 100,
    --check-source-regions 99).
  • npx prettier --check on both changed files — exit 0.

Local revision 900592b, node v26.10.0, TZ=UTC.

Coordination

Touches only main() (tests/tools/e2e-coverage-report.mjs) and appends to
tests/e2e-coverage-report.test.mjs.

Both append to tests/e2e-coverage-report.test.mjs at the end, so a trivial
append-order rebase is possible there; no logic overlaps.

Related Issue

Closes #1069

— hive: agent=quality backend=copilot model=claude-opus-5 copilot=1.0.88

main() in tests/tools/e2e-coverage-report.mjs built its failure report from
error.runId / error.runStatus, which only CoverageReportError carries. Every
throw raised after the manifest is read and validated is a plain Error, so the
published artifact header read

  E2E coverage: tooling-error (run null; status unknown)

even for a run sealed passed. CI cats that text straight into the step summary,
so a reader of a red End-to-end coverage job could not tell whether the
Playwright run failed or only the reporting step broke.

Fall back to the manifest, which readCoverageRun can re-read from --input. An
unreadable or absent manifest still yields null, exactly as before.

Closes #1069

Signed-off-by: quality <quality@hive.kubestellar.io>
@hivecommons-hive hivecommons-hive Bot added the hold label Oct 5, 2026
@hivecommons-hive

Copy link
Copy Markdown
Contributor Author

Important

Held for human review by the hive's ACMM level gate.

This PR was opened by the "quality" agent while Hive policy required a human checkpoint for that agent. Non-outreach agents are held at ACMM L3–L5; the outreach agent is always held because it publishes project-facing communication.

Hive will automatically remove the hold label once current policy no longer requires a level hold for "quality". If this is an outreach PR, a human must review it and remove the label.

…n-identity

Signed-off-by: quality <quality@hive.kubestellar.io>

# Conflicts:
#	tests/e2e-coverage-report.test.mjs
@mrbobbytables
mrbobbytables added this pull request to the merge queue Oct 5, 2026
@github-merge-queue
github-merge-queue Bot removed this pull request from the merge queue due to failed status checks Oct 5, 2026
@castrojo
castrojo added this pull request to the merge queue Oct 6, 2026
Merged via the queue into main with commit ce21b5d Oct 6, 2026
7 checks passed
@mrbobbytables
mrbobbytables deleted the quality/e2e-report-run-identity branch October 7, 2026 15:16
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[quality] e2e coverage failure reports say 'run null; status unknown' for every error raised after the manifest is read

1 participant