feat(gate): check that a lesson's cited source actually exists

[Ikalus1988]

User description

Why

The gates each cover one axis, and none of them covers truth:

gate	covers	catches an invented source?
scripts/lesson_gate.py	structure (fields, length, duplicate titles, domain)	❌
DCO	Signed-off-by	❌
scripts/injection_scan.py	prompt-injection shapes	❌

On 2026-09-16 four lesson PRs — #1713, #1714, #1715, #1716 — passed 24/24 checks while all four cited https://github.com/modelcontextprotocol/mcp-memory-service/issues/1652, with evidence_level: E3. That repository does not exist:

GET /repos/modelcontextprotocol/mcp-memory-service              -> HTTP 404
GET /repos/modelcontextprotocol/mcp-memory-service/issues/1652  -> HTTP 404

In a corpus whose value proposition is verifiable failure memory, an invented source is worse than an absent one: it looks checkable and is not. Today that class of pollution is caught only by a human reading every PR — which is exactly the bottleneck this removes.

What it does

scripts/check_provenance.py resolves the URLs a lesson cites in its frontmatter, in two tiers that reuse the strict-new/advisory split the lesson gate already uses (from #1506), so a 440-lesson legacy corpus cannot turn unrelated PRs red:

• tier 1 (always fails) — a placeholder URL (<owner>, TODO, {repo}, /xxx/, ...), or a URL confirmed dead (404/410) that is not recorded in the baseline.
• tier 2 (newly added files only) — evidence_level: E2/E3 while citing no resolvable source. Advisory, printed with the fix.

Deliberately conservative statuses: a timeout, DNS failure, rate limit (403/429) or 5xx is unknown and never fails the build, because a gate that goes red on someone else's outage gets bypassed within a week. Private-IP and example.com URLs are exempt by construction — a lesson about a corporate proxy has to be able to say http://172.19.128.1:7890, which is already in the corpus.

data/provenance-baseline.json is the debt register for known-dead links (why required per entry); placeholders can never be baselined — there is a test asserting that, because a gate that can be switched off is not a gate.

Verification

$ python3 scripts/check_provenance.py --check            # whole corpus
检查 439 篇课程，引用 31 条外链：none=408  ok=30  unknown=1
exit=0

# red-teamed against the real case, using #1713's lesson file verbatim
$ python3 scripts/check_provenance.py --check --strict-new /tmp/redteam/red.md
检查 1 篇课程，引用 1 条外链：dead=1
  ❌ source does not resolve — https://github.com/modelcontextprotocol/mcp-memory-service/issues/1652 (HTTP 404)
exit=1

$ python3 -m pytest tests/test_check_provenance.py -q
29 passed in 0.19s

Two bugs the tests caught while writing it, both now covered: --strict-new (nargs=*) swallowed the positional file list, so CI would have silently scanned the whole corpus instead of the PR's files; and a case-insensitive REPO placeholder pattern flagged any URL containing /repo/.

Boundaries (stated, not implied)

• It cannot check semantic truth: a real but irrelevant link passes. It removes the cheapest and most numerous class of pollution, not all of it.
• An exempt (private/example) source means "cannot be verified automatically", not "trustworthy".
• Prose claims with no link at all are out of scope.

Rules for contributors, including the three legitimate ways to satisfy it: docs/maintainer/provenance-gate-2026-09-16.md.

---

PR Type

enhancement, tests, documentation

---

Description

• New provenance gate verifies lesson citations resolve

• Two-tier rules: placeholder/dead links fail, high evidence without source warns

• Offline pytest suite covers 27 gate contract cases

• CI workflow, baseline register, and maintainer docs included

---

Diagram Walkthrough

flowchart TD
  A["Lesson frontmatter"] --> B["Extract URLs<br/>(source / evidence_refs / url)"]
  B --> C{"Classify<br/>ok / dead / unknown<br/>exempt / placeholder"}
  C -->|placeholder or dead| D{"In baseline?"}
  D -->|no| E["Tier 1: FAIL"]
  D -->|yes| F["Tier 1: pass"]
  C -->|unknown| G["Tier 1: pass<br/>(never fail on absence)"]
  C -->|ok or exempt| F
  C -->|new file + E2/E3 + none| H["Tier 2: advisory"]
  F --> I["CI: green"]
  E --> J["CI: red"]
  H --> I

Loading

File Walkthrough

	Relevant files
Enhancement	check_provenance.py New provenance gate checking lesson citations                        scripts/check_provenance.py Adds a new stdlib-only gate (urllib only) that scans every lesson frontmatter for source / evidence_refs / provenance / url / link URLs and classifies each as ok / dead / unknown / exempt / placeholder / none. Implements two tiers reusing the lesson-gate's strict-new/advisory split (from Lesson debt: legacy evidence_level (256) + gate new-vs-modified policy + en/ mirror dup FPs #1506): tier 1 always fails on placeholder URLs or confirmed-dead (404/410) links not in the baseline; tier 2 emits advisories when a newly-added file claims evidence_level: E2/E3 with no resolvable source. Maps github.com URLs onto the REST API for low-cost checks, reads GITHUB_TOKEN/GH_TOKEN, and treats timeouts / DNS / TLS / 403 / 429 / 5xx as unknown so the gate only fails on evidence, never on absence. Exempts illustrative addresses (localhost, example.com, RFC1918 ranges, *.local/*.internal/*.lan) by construction, and provides --check / --list / --offline / --strict-new / --update-baseline for CI and maintainer use; baseline I/O goes through data/provenance-baseline.json. +320/-0
Tests	test_check_provenance.py Offline tests for the provenance gate                                        tests/test_check_provenance.py Adds 27 offline pytest cases covering the gate's contract: a lesson with no citations produces no failure, placeholder sources fail without network, dead sources fail, and baselined dead links are tolerated. Pins the statuses that must never fail (unknown for 403/429/5xx/timeouts/DNS/offline, exempt for private-IP / localhost / example.com / *.local) and replays the exact feat(lesson): alembic upgrade failure diagnosis #1713 red-team URL as a regression test. Verifies tier-2 behaviour (high evidence_level without a source is advisory on new files only, silent on existing ones, and cleared by a resolvable URL), frontmatter parsing edge cases (list-form evidence_refs, GitHub→API mapping, missing frontmatter, malformed baseline), and CLI behaviour for --strict-new nargs="*" swallowing positional arguments. +222/-0
Documentation	provenance-gate-2026-09-16.md Maintainer docs for the provenance gate                                    docs/maintainer/provenance-gate-2026-09-16.md Documents the provenance gate's purpose using the 2026-09-16 incident where feat(lesson): alembic upgrade failure diagnosis #1713–feat(lesson): write success but data missing — silent failures #1716 passed 24/24 checks against a 404 GitHub repo, and shows why the existing three gates cannot catch invented sources. Explains the two-tier rule set (tier 1 always-fail for placeholders / dead links outside the baseline, tier 2 advisory for new files claiming E2/E3 with no source) and the status semantics, including why unknown must never fail. Lists exempt address patterns, baseline JSON shape with the "placeholders can never be baselined" invariant, a contributor self-help recipe (--list --check, three acceptable fixes), and the gate's stated limits (semantic truth, private sources, plain-text claims) plus next-step items already on the roadmap. +122/-0
Configuration changes	provenance-gate.yml Add provenance gate CI workflow                                                    .github/workflows/provenance-gate.yml New GitHub Actions workflow that runs scripts/check_provenance.py against changed lesson files on PRs Two-tier check: tier 1 (placeholder/dead URLs) always fails; tier 2 (newly added lessons with evidence_level: E2/E3 and no resolvable source) is strict-new PR runs are scoped to files the PR touches; weekly scheduled sweep + manual workflow_dispatch run against the full corpus as a report (continue-on-error) Posts a summary comment on the PR with the report and links to the rules doc, after first running the gate's unit tests +131/-0  provenance-baseline.json Add empty provenance baseline debt register                            data/provenance-baseline.json New JSON baseline file with schema misakanet-provenance-baseline/1 Empty known_dead and exempt_urls arrays (corpus currently passes without any entries) Inline note frames the file as a debt register for known-unresolvable sources, not a permission slip +6/-0

---

[Ikalus1988]

🧾 Audit Report — PR #1768 (35f5a24)

📊 Quality Score

⚠️ Quality score unavailable; continuing with hard gates.

🔏 DCO Audit

✅ All commits signed-off.

📏 PR Size

Metric	Value
Files Changed	5
Lines Added	801
⚠️ Warning	; 801 lines added (threshold: 500)

🔐 Secret Scan

✅ No hardcoded secrets detected.

📦 Dependency Audit

⏭️ Skipped; no Python/JS dependency files changed.

🧪 Test Suite

✅ PASS — 53% coverage

📋 Lesson Schema

✅ All lessons valid.

⚖️ Verdict

✅ All gates passed. Ready for merge.

---

Scope: full | Triggered by 35f5a24 | View run

[github-actions]

PR Reviewer Guide 🔍

Here are some key observations to aid the review process:

🎫 Ticket compliance analysis 🔶 1713 - Partially compliant Compliant requirements: None — this PR does not add the lesson content Non-compliant requirements: Add the lesson body covering Problem / Root cause / Solution / Prevention The cited solution playbook (stamp / merge / rollback) Requires further human verification: Whether this PR is expected to also close feat(lesson): alembic upgrade failure diagnosis #1713, or is purely a meta-gate that motivates off the lessons behind it 1714 - Partially compliant Compliant requirements: None — this PR does not add the lesson content Non-compliant requirements: Add the lesson body with proxy fix guidance (nginx/Caddy) and 30s stall monitoring prevention Requires further human verification: Whether this PR is expected to also close feat(lesson): SSE streaming failure via proxy or client parsing #1714 1715 - Partially compliant Compliant requirements: None — this PR does not add the lesson content Non-compliant requirements: Add the lesson body with the provider→model_id mapping table Requires further human verification: Whether this PR is expected to also close feat(lesson): Vertex AI vs Gemini model ID naming conventions #1715

⏱️ Estimated effort to review: 3 🔵🔵🔵⚪⚪

🧪 PR contains tests

🔒 Security concerns SSRF (defense-in-depth): The gate fetches any URL that is not matched by EXEMPT_PATTERNS and not classified as placeholder. The exempt list misses several non-routable / metadata ranges that a malicious lesson frontmatter could target, most importantly 169.254.0.0/16 (cloud instance metadata service 169.254.169.254), IPv6 loopback [::1], and IPv6 link-local fe80::/10. A lesson that cites http://169.254.169.254/latest/meta-data/iam/security-credentials/ will be fetched by the gate. Risk is currently low because lesson PRs are human-reviewed and CI runners are typically sandboxed, but a hardened gate should either reject such URLs outright or extend EXEMPT_PATTERNS to cover them. No other security concerns: token handling uses env vars only and is never logged, and SSL verification is left to urllib.

⚡ Recommended focus areas for review Incomplete EXEMPT_PATTERNS The EXEMPT_PATTERNS list covers RFC1918 IPv4 ranges (127/10/192.168/172.16-31) but does not cover link-local IPv4 (169.254.0.0/16, which includes the AWS/GCP/Azure instance metadata service at 169.254.169.254), IPv6 loopback ([::1]), or IPv6 link-local (fe80::/10). A lesson frontmatter citing such a URL is classified as check and the gate will issue an outbound HEAD/GET against it. In a CI runner with metadata-service access this is a defense-in-depth SSRF risk. The exempt block should be widened, or the gate should explicitly refuse (not silently fetch) any URL whose host resolves to a non-routable range. EXEMPT_HOSTS = {"localhost", "example.com", "example.org", "example.net", "test.invalid"} EXEMPT_PATTERNS = ( re.compile(r"^https?://(127\.|10\.|192\.168\.|172\.(1[6-9]|2\d|3[01])\.)"), re.compile(r"^https?://[^/]*\.(local|internal|lan)(:\d+)?(/|$)"), re.compile(r"^https?://[^/]*(localhost|127\.0\.0\.1)"), ) Case-sensitive OWNER/REPO/ORG regex PLACEHOLDER_PATTERNS includes \b(OWNER|REPO|ORG)\b as a case-sensitive match. GitHub usernames preserve case in URLs but are matched case-insensitively by the API, so a legitimate user https://github.com/OWNER/some-repo is technically valid. In practice such usernames are extremely rare (uppercase-only GitHub logins are nearly all taken by orgs/bots), so false positives are unlikely, but the rule will mis-flag any future real uppercase-only owner. Consider limiting this to template-shaped contexts (e.g., only inside path segments matching /OWNER/ after /github.com/). re.compile(r"\b(OWNER|REPO|ORG)\b"), # case-sensitive: placeholders are shouted Chinese CLI output text The summary line and warning/failure emojis are written in Chinese (检查 N 篇课程, 引用 ... 条外链, ⚠️). The rest of the file (docstrings, identifiers, argparse help) is English, and the tests assert against the Chinese strings (assert "检查 1 篇课程" in out). For a gate that runs on PRs from contributors who may not read Chinese, mixing locales is a maintenance hazard: future i18n changes will break tests and may confuse readers of CI logs. If bilingual output is intentional, at minimum add a note in the docstring; otherwise localize via a single constant. print(f"\n检查 {len({r['lesson'] for r in rows})} 篇课程，引用 {len([r for r in rows if r['url']])} 条外链：" + " ".join(f"{k}={v}" for k, v in sorted(counts.items()))) for note in advisories: print(f" ⚠️ {note}") for failure in failures: print(f" ❌ {failure}")

---

⚠️ Review coverage: The following files were not included in this review because of the token budget:

• docs/maintainer/provenance-gate-2026-09-16.md
• .github/workflows/provenance-gate.yml

[cloudflare-workers-and-pages]

Deploying with    Cloudflare Workers

The latest updates on your project. Learn more about integrating Git with Workers.

Status	Name	Latest Commit	Preview URL	Updated (UTC)
✅ Deployment successful! View logs	misakanet-web	9aefbf8	Commit Preview URL Branch Preview URL	Sep 16 2026, 03:26 PM

[github-actions]

PR Code Suggestions ✨

Explore these optional code suggestions:

Category	Suggestion	Impact
Possible issue	Comment step skipped when gate fails When the tier 1 step fails and exits 1, every subsequent step is skipped because the default condition only runs on success(). The friendly "Fix:" message is printed into the Actions log but never reaches the PR — defeating the entire purpose of this comment step. Add always() (combined with the existing pull-request guard) so the comment is posted regardless of whether the gate passed, failed, or was cancelled. .github/workflows/provenance-gate.yml [112-115] - name: Comment the result on the PR - if: github.event_name == 'pull_request' && steps.changed.outputs.any_changed == 'true' + if: always() && github.event_name == 'pull_request' && steps.changed.outputs.any_changed == 'true' continue-on-error: true uses: actions/github-script@3a2844b7e9c422d3c10d287c895573f7108da1b3 # v9 Suggestion importance[1-10]: 8 __ Why: The comment step has no explicit success condition, so GitHub Actions' default success() skips it whenever the tier 1 gate step exits non-zero. That defeats the PR-comment feedback loop the workflow is built around. Adding always() to the if expression is the correct fix.	Medium
General	Avoid duplicate PR comments on re-runs pull_request fires on every push to the PR branch (and on re-run), so each run appends a new top-level comment via createComment, quickly spamming the conversation with duplicates that all say the same thing. Make the comment idempotent: list existing comments, find one carrying a stable hidden marker, and edit it in place — only create a new comment when none is found. .github/workflows/provenance-gate.yml [126-131] - await github.rest.issues.createComment({ + const MARKER = '<!-- provenance-gate -->'; + const { data: comments } = await github.rest.issues.listComments({ issue_number: context.issue.number, owner: context.repo.owner, repo: context.repo.repo, - body: report, + per_page: 100, }); + const existing = comments.find(c => c.body && c.body.includes(MARKER)); + const body = MARKER + '\n' + report; + if (existing) { + await github.rest.issues.updateComment({ + owner: context.repo.owner, + repo: context.repo.repo, + comment_id: existing.id, + body, + }); + } else { + await github.rest.issues.createComment({ + issue_number: context.issue.number, + owner: context.repo.owner, + repo: context.repo.repo, + body, + }); + } Suggestion importance[1-10]: 6 __ Why: pull_request triggers on every push and re-run, so naively calling createComment each time will spam the conversation with identical comments. Using a hidden HTML marker and updateComment/listComments is a standard idempotency pattern; the proposed snippet is accurate and the concern is real, though less severe than a silently-skipped failure notification.	Low

⚠️ Suggestion coverage: 1 of 2 analysis chunks failed; the suggestions above are based on the successful chunks only.

[github-actions]

🎉 Merged — Thank you!

Your contribution has been merged into main.

PR: #1768 — feat(gate): check that a lesson's cited source actually exists

What's next:

• Your code is now part of MisakaNet's failure-lesson corpus (now 393 lessons)
• Feel free to pick up another issue labeled good first issue or status: competition
• Questions? Ask in this thread or open a Discussion

Welcome to the MisakaNet contributor community! 🧠

[github-actions]

✅ Merged! Thanks again, @Ikalus1988.

feat(gate): check that a lesson's cited source actually exists (+801 lines, 5 files)

Quick question — did any MisakaNet lesson help you this time?
→ Share feedback

No need to reply if nothing comes to mind. ⚡