Skip to content

[quality] add cross-catalog invariant tests — lock current state, catch new regressions - #3278

Merged
hivecommons-hive[bot] merged 1 commit into
masterfrom
quality/cross-catalog-invariants
Sep 11, 2026
Merged

hivecommons-hive[bot] merged 1 commit into
masterfrom
quality/cross-catalog-invariants

Conversation

@hivecommons-hive

Copy link
Copy Markdown
Contributor

Test Improvement

Adds scripts/__tests__/cross-catalog-invariants.test.mjs — 12 tests covering cross-catalog invariants that the mission generators depend on but that scripts/__tests__/catalog-invariants.test.mjs does not check.

Why

Two generators merge these three catalogs:

  • generate-cncf-missions.mjs:1441 — [...CNCF_PROJECTS, ...OTHER_PROJECTS.map(...)], keyed by .name, uses CATEGORY_TO_DIR[category] for the fixes/ subdirectory
  • generate-platform-missions.mjs:508 — [...K8S_PLATFORMS, ...OTHER_PROJECTS], keyed by .name

Silent violations of the merge assumptions corrupt generated output without failing CI today. The existing per-catalog suite plus one cross-cut (K8S ∩ OTHER by name) leaves the rest unguarded.

What this PR adds

Test group Invariant
Name collisions (generate-cncf-missions merge) CNCF ∩ OTHER by name, CNCF ∩ K8S by name (defensive)
Repo collisions CNCF internal repo uniqueness, CNCF ∩ OTHER by repo, K8S ∩ OTHER by repo
OTHER_PROJECTS pipeline compat .category ⊆ CATEGORY_TO_DIR keys, sources.stackoverflow.tags shape, sources.reddit.subreddits shape

While writing the suite, five distinct pre-existing data-debt violations surfaced on master (details filed as #3277). Rather than silently patch data or leave the suite red, each is captured in a named KNOWN_*_DEBT allowlist — the same pattern already used by catalog-invariants.test.mjs's KNOWN_TRAILING_HYPHEN_DEBT:

  • KNOWN_CNCF_OTHER_NAME_COLLISIONS — 6 entries
  • KNOWN_CNCF_K8S_NAME_COLLISIONS — 4 entries
  • KNOWN_CNCF_INTERNAL_REPO_DUPES — 1 entry
  • KNOWN_CNCF_OTHER_REPO_COLLISIONS — 5 entries
  • KNOWN_UNMAPPED_OTHER_CATEGORIES — 27 distinct categories

Each allowlist is paired with a companion ... stays honest test that fails if an allowlist entry no longer describes a real violation — so entries cannot be left in the list after the underlying debt is paid down.

Verification

$ npm test  # 'vitest run' in scripts/
Test Files  78 passed (78)
Tests       1146 passed | 1 expected fail (1147)

The new suite alone: Test Files 1 passed | Tests 12 passed. The full existing 1146-test suite still passes end-to-end.

Scope

  • No production data change. cncf-projects.mjs, k8s-platforms.mjs, other-projects.mjs are untouched.
  • No production code change. No generator or helper modified.
  • No workflow change. .github/workflows/scripts-tests.yml already runs npm test (= vitest run) which auto-discovers *.test.mjs files under scripts/__tests__/.
  • Complements — does not replace — the per-file invariants in scripts/__tests__/catalog-invariants.test.mjs.

Refs #3277


Filed by quality agent (ACMM L4/L6 — full mode)

— hive: agent=quality backend=copilot model=claude-opus-4.7

… new regressions

Adds scripts/__tests__/cross-catalog-invariants.test.mjs — 12 tests
covering the cross-catalog invariants that the mission generators
depend on but that scripts/__tests__/catalog-invariants.test.mjs
does not cover:

  * CNCF_PROJECTS ∩ OTHER_PROJECTS by name (silent clobber in
    generate-cncf-missions.mjs:1441 ALL_PROJECTS merge)
  * CNCF_PROJECTS ∩ K8S_PLATFORMS by name (defensive)
  * CNCF internal repo duplicates
  * CNCF_PROJECTS ∩ OTHER_PROJECTS by repo (double-crawl)
  * K8S_PLATFORMS ∩ OTHER_PROJECTS by repo
  * OTHER_PROJECTS.category ⊆ Object.keys(CATEGORY_TO_DIR)
  * OTHER_PROJECTS.sources.{stackoverflow.tags,reddit.subreddits}
    non-empty string[] when present

While writing the suite, five distinct current-state violations
surfaced on master (see #3277 for the full inventory). Rather than
silently patch the data or leave the suite red, each violation is
captured in a named KNOWN_*_DEBT allowlist so:

  * master stays green today
  * every NEW collision or unmapped category is caught immediately
  * companion 'allowlist stays honest' tests prevent bit-rot — an
    entry cannot be left in an allowlist after the underlying debt
    is fixed

No production data change; no workflow change (scripts-tests.yml
already runs 'vitest run' via 'npm test', which auto-discovers new
*.test.mjs files under scripts/__tests__/).

Refs #3277

Signed-off-by: kubestellar-hive[bot] <223556219+Copilot@users.noreply.github.com>
@hivecommons-hive

Copy link
Copy Markdown
Contributor Author

Important

Held for human sign-off on the direction, not on the code.

This PR's only tracked rationale is #3277, which the hive filed itself — issue #3277 was filed by kubestellar-hive[bot] and no human has acknowledged it. An agent-filed issue does not, on its own, establish that anyone agreed to the direction (hivecommons/hive#5117).

The change may well be right; nothing here is a review of it. To release the hold, acknowledge the direction on that issue — comment on it, assign yourself, or add the approved-direction label — and remove the hold label here.

@kubestellar-prow kubestellar-prow Bot added the dco-signoff: yes Indicates the PR's author has signed the DCO. label Sep 8, 2026
@kubestellar-prow

Copy link
Copy Markdown
Contributor

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please assign clubanderson for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@kubestellar-prow kubestellar-prow Bot added the size/L Denotes a PR that changes 100-499 lines, ignoring generated files. label Sep 8, 2026
@clubanderson clubanderson removed the hold label Sep 11, 2026
@clubanderson

Copy link
Copy Markdown
Member

Direction acknowledged by hub admin (approved-direction added on #3277); releasing the #5117 hold. This hive runs at ACMM L6 and may act on its own findings.

@hivecommons-hive
hivecommons-hive Bot merged commit 8a47133 into master Sep 11, 2026
8 of 11 checks passed
@kubestellar-prow
kubestellar-prow Bot deleted the quality/cross-catalog-invariants branch September 11, 2026 16:30
@github-actions

Copy link
Copy Markdown
Contributor

Thank you for your contribution! Your PR has been merged.

Check out what's new:

Stay connected: Slack #kubestellar-dev | Multi-Cluster Survey

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

copilot dco-signoff: yes Indicates the PR's author has signed the DCO. size/L Denotes a PR that changes 100-499 lines, ignoring generated files.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants