Skip to content

fix(ci): the docs auto-merge gate stops treating lessons as documentation - #1868

Merged
Ikalus1988 merged 1 commit into
mainfrom
fix/docs-gate-is-not-a-lesson-gate
Sep 19, 2026
Merged

Ikalus1988 merged 1 commit into
mainfrom
fix/docs-gate-is-not-a-lesson-gate

Conversation

@Ikalus1988

@Ikalus1988 Ikalus1988 commented Sep 19, 2026 •

Copy link
Copy Markdown
Owner

User description

Found while answering "what does it cost to loosen auto-merge-docs so it actually gets used?" — the
answer turned out to be "first fix this, or what you loosen is lesson content".

The hole

auto-merge-docs.yml merges external docs-only PRs to main, where docs.yml publishes them. Its
docs-only test was:

f.filename.endsWith('.md') || f.filename.endsWith('.txt') || f.filename.startsWith('docs/') || …

any .md includes lessons/** — directly under a comment stating that lessons/ is deliberately
not docs-only, because lesson content is read and acted on by agents.

It was armed

fact evidence
area:docs is applied automatically to every *.md .github/labeler.yml: area:docs ← **/*.md
external lesson PRs therefore satisfy the gate's label condition #1746, #1750: area:docs + area:lessons + lessons-only + needs-human-review
the gate also fires on synchronize / ready_for_review, i.e. after the label lands #1841: label at 08:51, ready_for_review at 15:17
what actually stopped them was DCO, not review #1846/#1746/#1750 all carry needs-dco

So an external lesson PR that is pushed again, or leaves draft, after the label appears reaches
gh pr merge --auto with docs-only: true.

The fix

  • the rule is one named function, stricter than before: docs/** or one of CONTRIBUTING.md /
    README.md / JOIN.md / CHANGELOG.md, and never lessons/. Stray .md/.txt anywhere else
    (scripts/README.md, fork-error.txt at the root) no longer qualifies either;
  • the job condition refuses PRs labelled lessons-only or needs-human-review;
  • tests/test_auto_merge_docs.py runs the real function over 14 paths — including
    lessons/contrib/x.md → false — and asserts the label refusals structurally.

This is deliberately the stricter direction. Note it separately means the gate still cannot fire on
the legitimately docs-only external PRs it was built for (#1842, #1801): they never re-trigger after
the label is applied, which is a trigger question, not a rule question — tracked in the issue comment
that prompted this.

  • Lesson content can no longer reach this path
  • The rule is stricter, named, and executed by a test
  • Label refusals added on top

PR Type

Bug fix, tests


Description

  • Fix CI docs auto-merge gate excluding lessons

  • Add isDocsFile rule with explicit lessons/ exclusion

  • Refuse PRs labelled lessons-only or needs-human-review

  • Add regression test running the real workflow function


Diagram Walkthrough

flowchart LR
  A["Old rule: any *.md is docs-only"] --> B["Lesson PRs auto-merge eligible"]
  B --> C["Poisoning risk: unreviewed agent-readable content"]
  D["New rule: isDocsFile excludes lessons/"] --> E["Adds label refusals: lessons-only, needs-human-review"]
  E --> F["Test runs real function over 14 paths"]
  F --> G["Lesson PRs require human merge"]
Loading

File Walkthrough

Relevant files
Bug fix
auto-merge-docs.yml
Tighten docs auto-merge rule to exclude lessons                   

.github/workflows/auto-merge-docs.yml

  • Replaced loose endsWith('.md') docs-only check with named isDocsFile()
    function
  • Added explicit !path.startsWith('lessons/') guard with explanatory
    note
  • Added job-condition refusals for lessons-only and needs-human-review
    labels
  • Tightened file allowlist to only docs/, CONTRIBUTING.md, README.md,
    JOIN.md, CHANGELOG.md
+27/-16 
Tests
test_auto_merge_docs.py
Add regression test for docs auto-merge gate                         

tests/test_auto_merge_docs.py

  • New structural test asserting lessons/ is excluded and label refusals
    are present
  • Behavioural test extracts rule from workflow markers and runs it via
    node -e
  • Covers 14 paths: docs, lessons, named files, and unrelated files
  • Skips node-based eval when node is unavailable
+110/-0 

…tion

`auto-merge-docs.yml` merges external contributors' docs-only PRs straight to main, where `docs.yml`
publishes them. Its docs-only test was `f.filename.endsWith('.md') || f.filename.endsWith('.txt') || …`
— true for **any** `.md` anywhere, `lessons/**` included, directly under a comment that says
"`lessons/` is deliberately NOT docs-only" because unreviewed lesson content is a poisoning vector.

That hole was armed, not theoretical:

* `.github/labeler.yml` gives `area:docs` to every `**/*.md`, so external lesson PRs satisfy the job's
  label condition — #1746 and #1750 carry `area:docs` + `area:lessons` + `lessons-only` +
  `needs-human-review`, and both are still open;
* the job also runs on `synchronize` and `ready_for_review`, so a lesson PR that is pushed again or
  comes out of draft *after* the label lands reaches `gh pr merge --auto` with `docs-only: true`
  (#1841 did exactly that at 15:17, label applied 08:51);
* the only thing that stopped all of them was `needs-dco` — a check about commit trailers, not review.

The rule is now one named function (`isDocsFile`), stricter than before: a path qualifies only if it
is under `docs/` or is one of the four named files, and `lessons/` is never documentation. That also
drops `*.md`/`*.txt` from anywhere else (`scripts/README.md`, a stray `fork-error.txt` at the root),
which the old rule accepted. The job condition additionally refuses PRs labelled `lessons-only` or
`needs-human-review` — belt and braces, since the label that lets a PR in is applied automatically.

`tests/test_auto_merge_docs.py` *runs* the real function (extracted from the workflow between two
markers) over a table of 14 paths, and checks the two label refusals structurally. Node executes it
because the rule runs inside Actions; where node is absent the behavioural half skips rather than
passing vacuously.

Signed-off-by: Ikalus1988 <136884451+Ikalus1988@users.noreply.github.com>
@cloudflare-workers-and-pages

cloudflare-workers-and-pages Bot commented Sep 19, 2026 •

Copy link
Copy Markdown
Contributor

Deploying with  Cloudflare Workers  Cloudflare Workers

The latest updates on your project. Learn more about integrating Git with Workers.

Status Name Latest Commit Preview URL Updated (UTC)
✅ Deployment successful!
View logs
misakanet-web e2e26cf Commit Preview URL

Branch Preview URL
Sep 19 2026, 12:17 PM

@github-actions

Copy link
Copy Markdown
Contributor

PR Genius Analysis

  • Risk Level: medium_risk
  • PR Size: +137/-16 (153 lines, medium)
  • Impact: 2 files changed (.github, tests)
  • Rules: 8 core + 5 repo-specific

Checklist

  • ci_passing (PENDING) — checks are still running
  • dco_signoff (PASS) — all commits signed
  • tests_updated (PASS) — 1 test file(s) changed
  • issue_reference (PASS) — linked issue found

Anti-Patterns Detected

  • workflow_change (medium)

Suggestions

  • Verify workflow syntax with act or push to a test branch.

@github-actions

Copy link
Copy Markdown
Contributor

PR Reviewer Guide 🔍

Here are some key observations to aid the review process:

🎫 Ticket compliance analysis 🔶

1746 - Partially compliant

Compliant requirements:

Non-compliant requirements:

  • (not applicable — the listed tickets describe lesson additions, not the CI gate fix this PR addresses)

Requires further human verification:

  • Confirm whether any "ticket" actually owns this CI fix; the three tickets provided are all about adding lessons and none describe the auto-merge gate bug. If unowned, no compliance check is required.
⏱️ Estimated effort to review: 2 🔵🔵⚪⚪⚪
🧪 PR contains tests
🔒 No security concerns identified
⚡ No major issues detected

@Ikalus1988

Copy link
Copy Markdown
Owner Author

🧾 Audit Report — PR #1868 (e2e26cf)

📊 Quality Score

⚠️ Quality score unavailable; continuing with hard gates.

🔏 DCO Audit

✅ All commits signed-off.

📏 PR Size

Metric Value
Files Changed 2
Lines Added 137

🔐 Secret Scan

✅ No hardcoded secrets detected.

📦 Dependency Audit

⏭️ Skipped; no Python/JS dependency files changed.

🧪 Test Suite

✅ PASS — 54% coverage

📋 Lesson Schema

✅ All lessons valid.

⚖️ Verdict

✅ All gates passed. Ready for merge.


Scope: full | Triggered by e2e26cf | View run

@Ikalus1988
Ikalus1988 merged commit b2a0f4c into main Sep 19, 2026
24 checks passed
@github-actions

Copy link
Copy Markdown
Contributor

🎉 Merged — Thank you!

Your contribution has been merged into main.

PR: #1868 — fix(ci): the docs auto-merge gate stops treating lessons as documentation

What's next:

  • Your code is now part of MisakaNet's failure-lesson corpus (now 393 lessons)
  • Feel free to pick up another issue labeled good first issue or status: competition
  • Questions? Ask in this thread or open a Discussion

Welcome to the MisakaNet contributor community! 🧠

@github-actions

Copy link
Copy Markdown
Contributor

✅ Merged! Thanks again, @Ikalus1988.

fix(ci): the docs auto-merge gate stops treating lessons as documentation (+137 lines, 2 files)

Quick question — did any MisakaNet lesson help you this time?
→ Share feedback

No need to reply if nothing comes to mind. ⚡

@github-actions

Copy link
Copy Markdown
Contributor

PR Code Suggestions ✨

Explore these optional code suggestions:

CategorySuggestion                                                                                                                                    Impact
Possible issue
Job-condition assertion is fooled by its own comment

The slice taken before runs-on: includes the explanatory comment block that already
names lessons-only and needs-human-review verbatim, so the assertions pass even if
those label checks disappear from the real if: expression. Restrict the search to
the lines GitHub Actions actually evaluates (between the if: key and runs-on:) so a
regression that drops the refusal conditions is caught.

tests/test_auto_merge_docs.py [79-85]

-    job_if = WORKFLOW.read_text(encoding="utf-8").split("runs-on:")[0]
+    body = WORKFLOW.read_text(encoding="utf-8")
+    # Only the expression GitHub Actions evaluates, not the YAML comment that
+    # already names these labels as a justification for them.
+    job_if = body.split("\n    if:", 1)[1].split("runs-on:", 1)[0]
     for refusal in ("lessons-only", "needs-human-review"):
-        assert refusal in job_if, (
+        assert f"'{refusal}'" in job_if, (
             f"the job condition must refuse PRs labelled '{refusal}': `area:docs` is applied "
             "automatically to every *.md by .github/labeler.yml, so the label alone is not a "
             "human checkpoint"
         )
Suggestion importance[1-10]: 7

__

Why: The job_if slice taken before runs-on: includes the explanatory comment block just added to the YAML, which already mentions lessons-only and needs-human-review verbatim. The assertion therefore passes even if those refusal conditions are dropped from the actual if: expression, defeating the regression test for the very issue this PR is fixing. Restricting the slice to the evaluated expression (and matching the quoted label form) makes the test actually verify what it claims to verify.

Medium

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant