Skip to content

ci: centralize conformance in cf-integration - #123

Merged
lucarlig merged 11 commits into
mainfrom
user/luca/cf-integration-conformance-ci
Sep 1, 2026
Merged

ci: centralize conformance in cf-integration#123
lucarlig merged 11 commits into
mainfrom
user/luca/cf-integration-conformance-ci

Conversation

@lucarlig

@lucarlig lucarlig commented Aug 31, 2026

Copy link
Copy Markdown
Contributor

Summary

  • replace repository-owned conformance orchestration with the published cf-integration@0.2.0 binary
  • keep only the on-demand Conformance workflow, make conformance, make conformance-bless, and structured baselines in this repository
  • let cf-integration ci prepare-image find the exact merge-revision artifact and package the prebuilt data-plane image
  • run strict modern client → modern server conformance through the external data plane
  • trigger the blocking check only when an authorized collaborator comments /conformance

The data-plane workflow no longer checks out or builds devtools, and its artifact orchestration scripts live in the published CLI from merged devtools PR #11.

Conformance command

cf-integration conformance run --client-era modern --server-era modern --lane external-data-plane --baseline-dir tests/conformance/baselines

make conformance-bless adds --bless; operational failures remain unblessable.

Validation

  • Actionlint
  • Make dry runs for conformance and conformance-bless
  • Local build of the conformance-prebuilt Docker target
  • data-plane formatting, Clippy, nextest, cargo-deny, and cargo-shear checks
  • live failure-path validation proving nonzero exit and unchanged baselines under --bless
  • published cf-integration 0.2.0 installed and prepared this PR exact merge-revision image successfully in GitHub Actions

CI timing

The latest data-plane CI run passed all ten checks in 2m44s, compared with 2m46s immediately before this change. The build/artifact job improved from 39s to 32s; stripping and uploading the reusable binary took 3s.

The latest conformance run completed in 4m00s, compared with 4m21s for the previous artifact-reuse run. Installing cf-integration 0.2.0 took 4s and exact-artifact image preparation took 14s. CI has not slowed down.

Current blocker

The new workflow and both external data-plane client/server harnesses execute correctly, but the blocking check intentionally remains red on the current upstream control plane. The run used control-plane main at ffcc0822bd014cbf8a62171eead957324da14155:

  • fixture-direct server conformance: 30 passed, 10 expected failures
  • external data-plane client conformance: 3 passed, 1 expected failure
  • routed server conformance: skipped and reported as an operational failure because POST /gateways returned 502 after the control plane received MCP 400 from the strict-modern fixture

IBM/mcp-context-forge#6485 is still open. It allows strict-modern registration, but routed conformance also needs complete resource sync, including test://static-text.

Signed-off-by: lucarlig <luca.carlig@ibm.com>
Signed-off-by: lucarlig <luca.carlig@ibm.com>
Signed-off-by: lucarlig <luca.carlig@ibm.com>
Signed-off-by: lucarlig <luca.carlig@ibm.com>
Signed-off-by: lucarlig <luca.carlig@ibm.com>
Signed-off-by: lucarlig <luca.carlig@ibm.com>
@lucarlig

lucarlig commented Sep 1, 2026

Copy link
Copy Markdown
Contributor Author

After this workflow reaches main, comment exactly /conformance on a PR to run the suite. GitHub will add an 👀 reaction and post the final result here. For PR #123 itself, use the transitional run-conformance label because issue_comment workflows are loaded from the default branch.

@lucarlig
lucarlig marked this pull request as ready for review September 1, 2026 08:30
Signed-off-by: lucarlig <luca.carlig@ibm.com>
Signed-off-by: lucarlig <luca.carlig@ibm.com>
Signed-off-by: lucarlig <luca.carlig@ibm.com>
Signed-off-by: lucarlig <luca.carlig@ibm.com>
Signed-off-by: lucarlig <luca.carlig@ibm.com>

@cafalchio cafalchio left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@lucarlig
lucarlig merged commit 29a2a81 into main Sep 1, 2026
11 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants