Skip to content

test: hold every committed e2e data overlay to the directory, not a list - #1100

Merged
castrojo merged 1 commit into
mainfrom
quality/test-overlay-integrity
Oct 6, 2026
Merged

castrojo merged 1 commit into
mainfrom
quality/test-overlay-integrity

Conversation

@hivecommons-hive

Copy link
Copy Markdown
Contributor

Test Improvement

Nothing in the repository reads tests/e2e/fixtures/data/ or
tests/e2e/fixtures/data-variants/ to find out what is committed there. The
two guards over committed overlays in tests/e2e-data-fixtures.test.mjs carry
hand-written file lists (lines 180-196 and 218-248), and
grep -n readdir tests/e2e-data-fixtures.test.mjs returns nothing.

applyOverlay is already strict about paths, so an overlay naming a field a
regenerated data file no longer has throws and npm run build:e2e:coverage
fails loudly. The failure mode with no symptom is the opposite one — an
overlay that applies cleanly and changes nothing:

  • a set writing the value the data already carries;
  • an append of a record a collector has since started producing itself.

The build succeeds, the coverage build compiles a document identical to the
real data, and the branch the overlay exists to reach silently stops being
reached. From CI that is indistinguishable from a branch that was never
covered.

This adds tests/e2e-data-fixture-integrity.test.mjs, which derives its cases
from the directory contents and holds every committed overlay to the weaker
property the directory alone can establish:

  1. both fixture directories hold at least one overlay (so the sweeps below
    cannot pass vacuously);
  2. every overlay names a data/<name>.json that exists;
  3. every overlay still applies to it — the build's own guard, run where the
    message names the overlay instead of arriving from inside a webpack loader;
  4. every overlay still changes the document it patches;
  5. every overlay's description names the source file whose arm it covers, so
    a stale overlay is traceable to the component that stopped needing it.

It deliberately does not duplicate the existing guards: those pin
applyOverlay's behaviour against overlay documents built inline, which a
directory sweep does not and should not do.

Verification

Run locally at 8afaa67:

  • npm run check — exit 0.
  • npm run test:unit — 1946 passed, 0 failed (1941 before, plus the five
    cases here).
  • npm run test:unit:coverage — the new file reports 100.00% lines and
    100.00% regions; src files unchanged at 100.00% / 99.92%.
  • Negative control, isolated. Replacing
    tests/e2e/fixtures/data-variants/awards.json with a no-op that sets
    verifiedAt to the value data/awards.json already carries leaves the four
    other cases green and fails exactly one:
    tests/e2e/fixtures/data-variants/awards.json: applies cleanly but patches nothing, so the branch it exists to reach is no longer reached. The same
    no-op leaves npm run test:unit green without this file. The probe was
    reverted; it is not part of the diff.

Files claimed

tests/e2e-data-fixture-integrity.test.mjs (new). Nothing else is touched.
Disjoint from every open hold-gated PR — in particular it does not edit
tests/e2e-data-fixtures.test.mjs (#1034 lines 178-215, #1078 lines 78-91),
tests/e2e/data-variants.spec.js (#1034), or
tests/tools/e2e-coverage-report.mjs (#1040, #1070). It adds no fixture, so it
does not collide with #1093's data-variants/metrics.json, #1065's
fixtures/data/case-studies.json, #1054's fixtures/data/community-people.json,
or #1098's data-variants/radar-reports.json.

Related Issue

Closes #1099

Folding the two hand-written per-branch lists into the directory sweep does
require editing the occupied tests/e2e-data-fixtures.test.mjs; that is
recorded in #1099's Coordination section as out of scope and left for once
#1034 and #1078 land. It is not part of what #1099 tracks, so merging this
leaves nothing open on it.


Filed by quality agent (hold-gated mode). Human review required.

— hive: agent=quality backend=copilot model=claude-opus-5 copilot=1.0.88

The two guards over committed overlays in tests/e2e-data-fixtures.test.mjs
carry hand-written file lists, and nothing in the repository reads
tests/e2e/fixtures/data/ or tests/e2e/fixtures/data-variants/ to find out what
is actually there.

applyOverlay already rejects an overlay naming a path the real data no longer
has, so that failure is loud. The one with no symptom is the opposite: an
overlay that applies cleanly and changes nothing -- a set writing the value
already present, an append of a record a collector has since started producing
-- compiles a document identical to the real data and silently stops reaching
the branch it exists for.

Adds a directory-derived guard in a new file: every committed overlay names a
data file that exists, still applies to it, still changes it, and still names
the source whose arm it covers. Replacing data-variants/awards.json with a
no-op that sets verifiedAt to its current value leaves npm run test:unit green
before this and fails it after.

Closes #1099

Signed-off-by: quality <quality@hive.kubestellar.io>
@hivecommons-hive hivecommons-hive Bot added the hold label Oct 5, 2026
@hivecommons-hive

Copy link
Copy Markdown
Contributor Author

Important

Held for human review by the hive's ACMM level gate.

This PR was opened by the "quality" agent while Hive policy required a human checkpoint for that agent. Non-outreach agents are held at ACMM L3–L5; the outreach agent is always held because it publishes project-facing communication.

Hive will keep the hold label until a human removes it. Operators can make a deliberate one-off release during an ACMM level change with release_level_holds=true, but level changes never release this hold automatically.

@castrojo
castrojo added this pull request to the merge queue Oct 6, 2026
Merged via the queue into main with commit c389355 Oct 6, 2026
7 checks passed
@mrbobbytables
mrbobbytables deleted the quality/test-overlay-integrity branch October 7, 2026 15:18
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[quality] a committed e2e data overlay that patches nothing has no symptom: the overlay guards enumerate files by hand

1 participant