Repository navigation
ci: pr-workflow E2E (agentgateway) flakes on the egress MITM/HTTPS tests #281
Description
Problem
The pr-workflow job E2E (agentgateway) flakes on the egress suites. On #273 (branch fork/271-convert-legacy-durable-layout, a change to ateom's restore path that touches no egress code), attempt 1 of run https://github.com/giantswarm/substrate/actions/runs/38150663168 failed the job with TestActorEgressMITMTrust, TestActorEgressHTTPSByHostnameMITM and TestActorEgressHTTPS (https://github.com/giantswarm/substrate/actions/runs/38150663168/job/114503346623, and the dependent e2e-test with it); attempt 2 on the same commit passed every job, and the same code was green on #277 the first time.
The MITM egress tests have a flake history on this line: the identity suite's trust-bundle rotation overlapping the HTTPS egress tests (the RotateEgressTrustPool row in FORK.md; kagent-dev/substrate cf3eb024 runs the identity suite in a separate step for the same reason). After the re-pin that row records the flake as gone; this run shows the egress tests still fail together on a rerun-fixable basis, so the cause is either back or another one.
Proposed solution
Read the failed attempt's logs for the three tests (x509 unknown authority during a rotation, or a timeout on the first HTTPS fetch of a fresh sandbox), and either isolate the identity suite from the egress tests in pr-workflow's E2E job as upstream does, or fix the shared state the three tests trip over. Until then a rerun is the workaround and every rerun is a 20-minute delay on a pull request.
Acceptance criteria
- The cause of the three failures on attempt 1 is named in this issue from the job log.
- Ten consecutive
E2E (agentgateway)runs ongiantswarmpass without a rerun, or the pattern no longer appears on pull requests that touch no egress code.
Written by an agent.
Activity
teemow commentedon Oct 12, 2026
Seen again on 2026-10-12 (an agent wrote this), with the failure text this time: the egress tests (
TestActorEgressHTTPSand the MITM tests) fail withGet https://example.com/: EOF(HTTP 502) on https://github.com/giantswarm/substrate/actions/runs/38203936804 (a docs-only pull request, the re-run passed) and on the main push run https://github.com/giantswarm/substrate/actions/runs/38197307787. The tests depend on HTTPS egress to the third-party host example.com, so a slow or refusing upstream turns unrelated pull requests red at the merge gate. Besides the identity-suite overlap above, the fix this points at is a hermetic target: an in-cluster HTTPS server (its own CA, trusted through the egress trust bundle) the egress tests fetch instead of example.com.