Repository navigation
test: run ITS pipeline e2e checks on pull requests - #3574
dheerajodha wants to merge 1 commit into
Conversation
|
No actionable comments were generated in the recent review. 🎉 ℹ️ Recent review info⚙️ Run configuration
📒 Files selected for processing (1)
Included review availability: This review used your included allowance. Your plan provides up to 12 included reviews per hour; 11 remain after this review. 📝 WalkthroughWalkthroughAdds a Tekton PipelineRun for pull requests targeting ChangesCLI integration tests
Priority: ⬇️ Low Estimated code review effort: 2 (Simple) | ~10 minutes Change: Feature Suggested reviewers: Merge Risk: 🟡 Moderate · up to Confirm the tenant’s fork-approval policy before merging. Fork-sourced tests run with a registry credential available in the test environment, so authorization is a material prerequisite. 🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches 💡 1🧪 Generate unit tests (beta)
🛠️ Fix failing CI checks 💡
Comment |
|
Risk Assessment: low (1/5) DetailsSingle new PaC trigger YAML (53 lines, small blast radius), no protected paths, no security-sensitive files, no dependency changes, config-only PR treated as neutral on test ratio, returning contributor; new-file Tier 2 baseline of 2 nudges the composite to 1.46, which rounds to 1. Previous runRisk Assessment: moderate (2/5) DetailsRe-review anchoring preserves prior moderate (2) score: Tier 1 signals are unchanged (single new .tekton/ file, 53 lines, small blast radius, returning contributor, no protected paths, no dependencies), Tier 2 defaults to 2 for a new file, and the blocking cross-repo dependency (conforma/e2e-tests#12) that the prior assessor flagged remains unmerged — no signal justifies lowering from the prior score. Previous run (2)Risk Assessment: moderate (2/5) DetailsSingle-file Tekton CI pipeline addition (57 lines) by a returning contributor with small blast radius and no security concerns per Tier 1 signals; fork pins remain unresolved but debug tag was removed since prior assessment, keeping composite steady at moderate. Previous run (3)Risk Assessment: moderate (2/5) DetailsSmall config-only PR (59 lines, 2 files) adding a new CI trigger with a deliberately-broken bundle tag and fork-pinned refs; the debug-tag and fork pins are acknowledged pre-merge restorations, and stable git history plus returning contributor keep composite at moderate. Previous run (4)Risk Assessment: low (1/5) DetailsVery small PR (2 files, 63 lines added, blast=small) with no protected paths, security-sensitive files, dependency changes, or CI workflow modifications by the Tier 1 script's classification. Author is a known human contributor. Primary risk factors are draft status and an unmerged external dependency (e2e-tests#12), which introduce coordination coupling but do not affect the code risk of the change itself. Composite T1=1.1, T2=1.1, T3=2 yields 1.29, rounding to 1 (low), consistent with the PR's own risk/low label. Previous run (5)Risk Assessment: low (1/5) DetailsSingle small Tekton PipelineRun config file (54 lines) added under .tekton/. No protected paths, no security-sensitive content per the metadata script, no dependency changes, no GitHub Actions workflow changes. Author is a known human contributor. Tier 2 applies the all-files-new moderate baseline of 2. Weighted composite (Tier 1 62% x 1.125 + Tier 2 38% x 2) = 1.46, rounds to 1 (low). Note: this baseline score does not reflect the fork-code-execution risk raised as a high finding, which is qualitative and outside the Tier 1 signal set. |
ReviewFindingsHigh
Next steps:
Previous runReviewSince the prior review (SHA FindingsHigh
Medium
Low
Next steps:
Previous run (2)ReviewSince the prior review (SHA FindingsHigh
Medium
Low
Next steps:
Previous run (3)ReviewSince the prior review (SHA FindingsCritical
High
Medium
Low
Next steps:
Previous run (4)ReviewThe CEL author_association gate added since the prior review ( FindingsCritical
High
Medium
Low
Next steps:
Previous run (5)ReviewFindingsHigh
Next steps:
|
![fullsend-ai-review[bot]](https://avatars.githubusercontent.com/in/3478501?s=60&v=4){kind=small}
This comment was marked as outdated.
This comment was marked as outdated.
Sorry, something went wrong.
|
/retest |
Codecov Report✅ All modified and coverable lines are covered by tests.
Flags with carried forward coverage won't be shown. Click here to find out more. 🚀 New features to boost your workflow:
|
|
🤖 Review · Commit: |
|
🤖 Review · Commit: |
|
/retest |
|
Caution There are some errors in your PipelineRun template.
|
|
🤖 Finished Review · ✅ Success · Started 1:46 PM UTC · Completed 1:57 PM UTC Commit: Runtime: claude · Model: opus → claude-opus-4-7 · Effort: high · Cost: $4.89 |
This comment was marked as outdated.
This comment was marked as outdated.
Sorry, something went wrong.
|
🤖 Finished Review · ✅ Success · Started 2:02 PM UTC · Completed 2:12 PM UTC Commit: Runtime: claude · Model: opus → claude-opus-4-7 · Effort: high · Cost: $4.40 |
fullsend-ai-review
Bot
added
risk/moderate
This comment was marked as outdated.
This comment was marked as outdated.
Sorry, something went wrong.
|
🤖 Finished Review · ✅ Success · Started 2:48 PM UTC · Completed 2:58 PM UTC Commit: Runtime: claude · Model: opus → claude-opus-4-7 · Effort: high · Cost: $3.92 |
This comment was marked as outdated.
This comment was marked as outdated.
Sorry, something went wrong.
|
🤖 Review · Commit: |
|
🤖 Finished Review · ✅ Success · Started 12:11 PM UTC · Completed 12:22 PM UTC Commit: Runtime: claude · Model: opus → claude-opus-4-7 · Effort: high · Cost: $4.50 |
This comment was marked as outdated.
This comment was marked as outdated.
Sorry, something went wrong.
st3penta
left a comment
st3penta
left a comment
There was a problem hiding this comment.
lgtm. remember to update the references to the upstream repo before merging!
| resolver: git | ||
| params: | ||
| - name: url | ||
| value: https://github.com/dheerajodha/conforma-e2e-tests.git |
There was a problem hiding this comment.
I'm assuming this is WIP, right?
There was a problem hiding this comment.
In a way, but it's still ready for review. I was just waiting for the conforma/e2e-tests#12 to get merged first. And then update the refs here.
|
Looks like it's still WIP, consider moving to draft. |
|
🤖 Review · Commit: |
Run the ITS E2E suite for changes to the enterprise-contract pipeline or its CI trigger. Pin the runner and tests to the merged upstream E2E commit, and resolve the pipeline under test from the pull request source and revision. Refs: EC-1943
8e0348b to
4fc0222
Compare
fullsend-ai-review
Bot
added
risk/low
| - name: deprovision-aws-credentials-secret | ||
| value: mapt-kind-secret | ||
| - name: its-pipeline-repo-url | ||
| value: '{{source_url}}' |
There was a problem hiding this comment.
[high] fail-open
its-pipeline-repo-url (line 36) and its-pipeline-revision (line 38) interpolate PR-controlled {{source_url}}/{{revision}} and feed a PR-controlled Tekton PipelineRun/Task YAML to the e2e-tests runner, which executes it under serviceAccountName: konflux-integration-runner (line 53) with references to oci-container-repo-credentials-secret: konflux-test-infra (line 30), aws-credentials-secret: mapt-kind-secret (line 32), and deprovision-aws-credentials-secret: mapt-kind-secret (line 34). A fork PR could submit arbitrary Tekton task definitions to be executed in-cluster with that SA token and the mounted secrets. The only in-manifest authorization boundary is PaC default ACL (/ok-to-test); the PR body itself lists verification of the deployed PaC Repository/global policy as an unresolved checklist item. The upstream runner at conforma/e2e-tests@eb59162d could not be inspected from this review, so whether the PR-controlled ITS definition is sandboxed from the shared SA/secrets cannot be confirmed here.
Suggested fix: Before merge: (1) confirm with the tenant admin that the deployed PaC Repository/global policy enforces /ok-to-test for unauthorized contributors (already an open PR checklist item); and (2) inspect .tekton/pipelines/conforma-e2e/pipeline.yaml at the pinned SHA and verify the PR-controlled ITS definition is sandboxed from the three secrets above (separate TaskRun/namespace, or not mounting konflux-test-infra/mapt-kind-secret on tasks that execute PR-controlled YAML). If either prerequisite cannot be verified, scope the SA/secrets to the minimum the outer runner needs and remove references the PR-controlled stage can reach.
|
The Red Hat Konflux / cli-its-on-pull-request check failed due to some Kubernetes dependency issue, it's previous runs were successful, I'm looking into it now. |
Opened PR: conforma/e2e-tests#33 |
This branch has not been deployed
4 participants
Changes to the enterprise-contract ITS pipeline need pre-merge E2E validation. Add a separate Pipelines-as-Code check for PRs targeting main that change
pipelines/enterprise-contract/**or this trigger. It runs only the ITS suite and reports the result on the CLI PR.The runner and test suite are pinned independently of the pipeline under test. The ITS definition is fetched from the PR source URL and exact commit SHA, including fork PRs. The definition retains its own task bundle references; this check does not build a custom CLI image.
Dependency and authorization
Dependency conforma/e2e-tests#12 is merged. Both runner and test-suite sources now use
https://github.com/conforma/e2e-tests.gitpinned to its upstream merge commiteb59162d7c1d069a699f82a50e9f41db852c53d2.Matching uses normalized event, target branch, and changed paths. Execution authorization relies on Pipelines-as-Code ACLs and approval policy. Verify the tenant's deployed Repository/global authorization configuration before merge; it has not yet been inspected. A raw-webhook author-association filter was removed because event payload differences prevented execution. The outer runner is pinned to a fixed E2E commit; only the ITS definition under test uses the PR source URL and revision. Pipelines-as-Code documents ACL checks before execution, including
/ok-to-testapproval for unauthorized contributors: https://pipelinesascode.com/docs/guides/running-pipelines/#acl-permissions-for-triggering-pipelineruns. An author filter inside a PR-editable trigger is not a substitute for that external authorization boundary.Validation
The PR history is now one signed commit,
4fc0222b, based on upstream maind7152fd9. The final diff adds only the ITS trigger; its contents are unchanged by the rebase. YAML parsing, upstream runner parameter checks, and whitespace validation passed. Fresh CI results are pending. The runs below document earlier positive and negative validation.cli-its-on-pull-request-t6hlvtested CLI commit257252cb; all three ITS scenarios passed.cli-its-on-pull-request-v6kmvtested CLI commitb6987722. Image building, signing, and attestation succeeded. The ITS PipelineRun then failed withCouldntGetTaskandMANIFEST_UNKNOWNfor the deliberately nonexistent bundle tagec-1943-deliberately-missing-round2. The success scenario failed and the E2E step exited 1; the other two ordered scenarios were skipped.cli-its-on-pull-request-sq7brpassed against CLI commit455530c490a3a4f4a79a54a60dd220730a31f687, with the validquay.io/conforma/tekton-task:konfluxreference restored.6ead7e45: every supplied parameter name/type matches the runner at4bbba993; both runner/test source pins agree; no deliberate-break bundle tag remains. The PR changes only the trigger file.Negative-run logs: https://konflux-ui.apps.stone-prd-rh01.pg1f.p1.openshiftapps.com/ns/rhtap-contract-tenant/pipelinerun/cli-its-on-pull-request-v6kmv/logs/conforma-e2e-tests
Before merge
4fc0222b, rebased onto upstream maind7152fd9).Remaining rollout work
Complete required-check enforcement without leaving unrelated PRs waiting for a path-filtered check. Keep EC-1943 open until merge protection is active. Separately investigate the artifact collection permission errors seen in the negative run; they did not cause the confirmed bundle-resolution failure.