test: add Python MCP conformance CI

[lucarlig]

Pull Request

🔗 Related Issue

closes #6018

📝 Summary

Adds official MCP 2025-11-25 and 2026-07-28 conformance for the gateway server and outbound client paths using the control plane and built-in Python data plane only. Includes make conformance, make conformance-bless, expected-failure baselines, and PR CI. The isolated harness disables rate limiting/account lockout.

---

📏 Reviewability

• One clear purpose
• No linked triage issue (none linked)
• No unrelated work
• Tests included

---

🏷️ Type of Change

• Bug fix
• Feature / Enhancement
• Documentation
• Refactor
• Chore (deps, CI, tooling)
• Other

---

🧪 Verification

Check	Status
make conformance	✅ Passed
Reporter and shell-script tests	✅ Passed
Current-head PR checks	✅ All passed
MCP 2025 + 2026 CI job	✅ 1m 30s

---

✅ Checklist

• Formatting/hooks pass
• Tests added/updated
• Commands exposed through make help
• No secrets committed

---

📓 Notes

Revision	Pass	Expected failure	Upstream excluded	Unexpected
2025-11-25	45	13	0	0
2026-07-28	8	89	6	0

Complete MCP 2025-11-25 expected-failure inventory

Scenario/check	Count	Why it fails
dns-rebinding-protection:localhost-host-rebinding-rejected	1	Invalid Host/Origin is accepted with HTTP 200 instead of rejected with 4xx.
elicitation-sep1034-defaults, elicitation-sep1330-enums	2	The gateway does not relay the fixture's elicitation request to the client.
prompts-get-embedded-resource, prompts-get-with-image	2	The gateway returns []; the client expects a valid GetPromptResult.
resources-read-binary	1	Returned resource content is missing the required blob field.
resources-templates-read	1	The instantiated test://template/123/data resource is reported as not found.
resources-subscribe, resources-unsubscribe	2	Both operations return JSON-RPC Method not found.
server-sse-multiple-streams:server-sse-multiple-streams-session	1	The runner warns because the server supplies a session ID in the multiple-streams case.
tools-call-elicitation	1	No elicitation request reaches the client during tool execution.
tools-call-sampling	1	No sampling request reaches the client during tool execution.
tools-call-with-logging	1	No log notification reaches the client.

The former valid-localhost, prompt-list, simple-prompt, and argument-prompt 429 findings now pass after disabling harness rate limiting.

Complete MCP 2026-07-28 expected-failure inventory

ContextForge currently supports protocol revisions through 2025-11-25. It rejects 2026-07-28 with HTTP 400 and a plain error object rather than a JSON-RPC error. The table accounts for all 89 scored failures; dependent capability and wire-schema checks fail from that same version-negotiation response.

Scenario group	Count	Why it fails
caching:*	7	Version rejection prevents list endpoints from returning SEP-2549 caching hints; the rejection body also fails wire-schema validation.
completion-complete:*	2	completion/complete is rejected before execution; the HTTP error body is not valid JSON-RPC.
dns-rebinding-protection:localhost-host-valid-accepted	1	Even valid localhost headers receive HTTP 400 because the requested protocol revision is unsupported.
input-required-result-* (basic elicitation, list-roots, sampling, capability check, extra params, missing response, multi-round, multiple requests, non-tool request, request state, result type, tampered state, unsupported methods, validation)	26	Version rejection prevents SEP-2322 InputRequiredResult flows; each affected response also fails the negotiated wire schema.
prompts-list, prompts-get-simple, prompts-get-with-args, prompts-get-with-image, prompts-get-embedded-resource	10	Prompt requests are rejected for the unsupported revision; each functional check and wire-schema check fails.
resources-list, resources-read-text, resources-read-binary, resources-templates-read, sep-2164-resource-not-found	11	Resource requests are rejected before capability behavior is exercised; functional and wire-schema checks fail.
server-sse-multiple-streams:server-accepts-multiple-post-streams	1	The additional stream request is rejected during 2026 protocol validation.
server-stateless:* (all scored SEP-2575 checks)	23	Stateless discovery, _meta, capability, method/error, subscription, and header requirements are not implemented for 2026; rejection responses also omit required JSON-RPC error fields/IDs.
tools-call-simple-text, tools-call-image, tools-call-audio, tools-call-embedded-resource, tools-call-mixed-content, tools-call-error, tools-list	8	Tool requests are rejected for the unsupported revision. Six functional call checks fail, while tools/list fails both behavior and wire-schema validation.

Excluded or unscored fixture findings

Scenario	Status	Explanation
Six tools-call-*:wire-schema-valid checks in 2026	Upstream excluded	The pinned official JavaScript fixture reproduces these six failures when tested directly, so they are not attributed to ContextForge.
tools-call-with-progress	Temporarily excluded	The pinned progress scenario is flaky; its 2025/2026 baseline and upstream entries remain recorded for re-enablement.
2025 server-sse-polling	Unscored/excluded	Official requirements mark it pending; excluding its fixed 60-second timeout does not change the score.

CI builds the PR merge commit from Containerfile, runs the resulting local image in Python mode, and never pulls a ContextForge latest image or builds Rust. The conformance job takes 1m 30s and runs alongside pytest.

[msureshkumar88]

Reviewed the full diff (via the GitHub API, not the summarized CLI diff) against main and traced the surrounding code the PR depends on rather than reviewing the diff in isolation.

Scope: purely CI/test-harness — 21 files under .github/workflows/, tests/conformance/, Makefile, .gitignore, .secrets.baseline. No mcpgateway/ application code touched, no Alembic migration needed, not a UI or backend change.

Verified, not just trusted:

• RATE_LIMITING_ENABLED=false in the new compose file correctly maps to Settings.rate_limiting_enabled (mcpgateway/config.py) and actually disables RateLimitMiddleware enforcement including its lockout path (rate_limit_middleware.py) — the "disable rate limiting" fix commit isn't a no-op.
• All referenced Containerfile build args, the /health route, and MCPGATEWAY_SKIP_MIGRATIONS exist as expected.
• Pinned GitHub Action SHAs match what's already used elsewhere in the repo's workflows.
• Cross-checked every count in the PR description's expected-failure tables against expected-failures-2025-11-25.yml / expected-failures-2026-07-28.yml line by line — all match exactly.
• report-baseline-diff.sh (the custom baseline-diff/bless logic) has solid dedicated test coverage in report-baseline-diff-test.sh covering xfail/xpass/upstream-exclusion/duplicate/bless/missing-results paths.

No blocking issues. A couple of non-blocking suggestions for a fast-follow, not required for this PR:

• tests/AGENTS.md could document the new tests/conformance/ directory and the make conformance / make conformance-bless commands.
• Consider adding a "test-only, do not reuse" banner comment to tests/conformance/docker-compose.yml, since it intentionally disables SSRF protection and rate limiting and uses a hardcoded (allowlisted) password — reasonable for this isolated, ephemeral harness, but worth flagging so it's never lifted as a starting template elsewhere.

Nice addition — this wires the project into the official, versioned MCP conformance suite for both the current and next protocol revision, with a maintained expected-failure baseline that makes real protocol/behavioral gaps visible in CI instead of undocumented.

[msureshkumar88]

Re-reviewed after the latest commit (529e963). It's local-output UX polish only — progress spinner, quieter setup logs, a run lock, and renaming expected-failures.yml → baseline.yml with matching test/reporter updates in report-baseline-diff.sh / report-baseline-diff-test.sh. No mcpgateway/ application code touched.

My earlier review had no blocking findings, just two non-blocking suggestions for a fast-follow (documenting tests/conformance/ in tests/AGENTS.md, and a "test-only" banner on the compose file since it disables SSRF protection/rate limiting) — neither is required for this PR.

One heads-up: GitHub shows this branch as conflicting with main (just .secrets.baseline line-number churn and non-overlapping Makefile additions) — worth a quick rebase before merge, unrelated to review content.

Approving.