Skip to content

fix(e2e): make nightly results trustworthy and unblock iOS on RN 0.82 - #5113

Merged
moskalakamil merged 9 commits into
masterfrom
ci/e2e-nightly-first-run-fixes
Sep 18, 2026
Merged

fix(e2e): make nightly results trustworthy and unblock iOS on RN 0.82#5113
moskalakamil merged 9 commits into
masterfrom
ci/e2e-nightly-first-run-fixes

Conversation

@mziolkowskii

@mziolkowskii mziolkowskii commented Sep 17, 2026 •
edited
Loading

Copy link
Copy Markdown
Collaborator

Summary

The first e2e-nightly runs on master were red for reasons that have nothing to do with the library: 9 issues (#5104–#5112) from one night, three of them false. This PR fixes the harness and CI so that a red nightly row means something. No library code is touched.

# Problem Cause Fix
1 All iOS rows missing-report, even green legs (#5107, #5111, #5112) An upload path outside the workspace (~/.maestro/tests/**) moved the artifact root, so the report job did not find e2e-report.xml Copy those logs into the workspace first (_e2e-ios.yml)
2 Every API 34 leg red, random flow on a white screen (#5104–#5106) Android 14 kills the package's process ~1 s after pm clear ("remove task"); launchApp: {clearState: true} starts the app inside that window clearState, 2.5 s wait (Android only), launchApp
3 Job summary never showed why a flow failed Maestro 2.10 writes the reason as element text, the script read a message attribute junit-summary.mjs reads both
4 One iOS leg lost 10/10 flows to a leftover "Open in RNVideoE2E?" prompt The prompt appeared after the warm-up stopped waiting; it is SpringBoard's, survives relaunches and hides the app launch-app.yaml answers a prompt that is already up, then relaunches
5 iOS never built on RN 0.82 (#5109, #5110) fmt consteval errors when NitroModules' Swift compile builds the fmt module as C++20, and behind that a glog module error (include inside a namespace) Podfile post_install patches fmt/base.h and glog.modulemap
6 A simctl openurl timeout failed a flow; the retry written for it never ran Maestro's retry/optional only catch MaestroException; this is an IllegalStateException iOS opens deep links through the fixture server, which retries simctl

Details, logs and measurements are in the individual commit messages; e2e/CONTEXT.md gets a gotcha for 2, 4 and 6.

Where to look

Two changes deserve a real review, the rest is mechanical:

  • e2e/fixtures/open-link.mjs + the /__open-link route in serve.mjs. The fixture server now runs a process on request. It answers loopback only, accepts nothing but rnvtest://scenario/<name> and a simulator UDID (both regex-validated), and calls xcrun through execFile, never a shell. It retries the transport only: what the app shows is still asserted once (D11 holds). A link delivered twice is harmless because the test app keys its screen on the scenario; that was tested, see below. Android keeps Maestro's openLink, and the retry around it is removed rather than kept as decoration.
  • test-app/ios/Podfile. post_install edits two files inside Pods/. Both edits are idempotent and skipped when the file is absent. The module map's mtime is pinned, because CocoaPods regenerates it on every pod install and a changed mtime would defeat the DerivedData cache. To be removed once RN ships an fmt that builds on Xcode 26 and Nitro stops pulling glog into its umbrella.

Cost: about 4 s per Android flow and about 7 s per iOS flow in extra waits. "Run flows" durations did not grow against runner variance (314–535 s after, 359–643 s before, same legs).

Verification

Nightlies dispatched from this branch (they write to e2e-results and manage the nightly issues themselves, which is why this PR has no "Fixes #"):

Run Contains Result
35198102655 fixes 1–3 Android 9/9 (API 34: 3/3, was 0/3). iOS rows carry real reports. The report job closed #5104–#5108 and #5112.
35204573155 + fix 4 Android 9/9, the leg from problem 4 green. Left: RN 0.82 build, one simctl timeout.
35212563832 + fixes 5, 6, review fixes 15/15 rows green, 150/150 flows. Both RN 0.82 iOS legs built and passed for the first time; RN 0.77 iOS builds with the new Podfile. The report job closed the last three issues (#5109, #5110 and the reopened RN 0.77 / iOS 26 row); no e2e-nightly issue is open.

Reproduced locally before fixing (Xcode 26.6, iOS 26.5 simulator, API 34 emulator):

  • 2: pm clear; sleep D; am start, 15 launches each: 2 killed at D=0.9 s, 0 at D=2.5 s.
  • 4: an unanswered prompt makes the old launch-app.yaml fail exactly as in CI; the new one passes and later links open without a prompt.
  • 5: the NitroModules target fails with the CI errors before the change; after it the whole workspace builds and the app runs.
  • 6: warm-up and three flows pass through the new transport. With an xcrun wrapper that delivers the link and then reports a timeout, every open takes two attempts and the flows still pass. With the server down, a flow fails with a clear open-link failed: … localhost:8090.

Unit tests: 188 pass, 22 new (link/UDID validation, retry, the route against a fake xcrun, junit-summary edge cases). bun lint and bun typecheck pass.

Risk and rollback

Nothing here ships to users. Each fix is its own commit and reverts cleanly on its own. The least proven one is 2: it rests on one logcat analysis plus the adb reproduction, so a few more nights will tell. RN 0.77 on iOS with the new Podfile was not built locally; run 35212563832 built and passed it on both runners.

Not addressed: two lost taps on RN 0.77 / iOS 26 from the first night (#5108, closed itself on the next green run), and the prompt check in 4 runs once per launch, so a prompt that appears later still costs that one flow.

Platforms affected

  • Android
  • iOS
  • visionOS
  • tvOS / Android TV
  • Windows
  • Web

Type of change

  • Bug fix (non-breaking change that fixes an issue)
  • New feature (non-breaking change that adds functionality)
  • Breaking change (fix or feature that changes existing behavior)
  • Documentation only

Checklist

  • I read the contributing guidelines.
  • I added or updated tests that cover this change, or explained under "Test plan" why none apply.
  • For a bug fix the E2E suite can exercise, this PR includes a Maestro flow that fails without the fix, or "Test plan" explains why none does. (None: harness, CI and build defects, not library behaviour.)
  • bun run test, bun lint and bun typecheck pass locally.
  • I updated the documentation / README where relevant.
  • If this changes how the library is used (API, props, events, behavior), I updated the AI agent skill (skills/react-native-video/) to match. (Not applicable.)
  • This PR is focused on a single concern. (One theme, six fixes: they were found by verifying each other on the nightly, and the later ones only became visible once the earlier ones were in. Each is a separate commit.)
  • I tested my changes on at least one platform, or described under "Test plan" what CI (which runs both) should show.

@mziolkowskii
ci(e2e): keep the iOS artifact rooted at the workspace so nightly fin…
e91d1b9 
…ds the report

The first nightly on master (run 35173748419) scored all six iOS rows as
`missing-report` and opened an issue for each, including the three legs that
passed 10/10 (#5107, #5111, #5112).

upload-artifact roots the archive at the least common ancestor of the matched
paths. `~/.maestro/tests/**` is outside the workspace, so the root became
/Users/runner and e2e-report.xml was stored at
work/react-native-video/react-native-video/e2e-report.xml. The report job reads
reports/<label>/e2e-report.xml, found nothing, and record-result.mjs correctly
treats a missing report as a failure. Android uploads only workspace paths and
was unaffected.

Copy ~/.maestro/tests (the warm-up flow's logs and the XCTest runner log) into
the workspace and upload it from there.
@mziolkowskii
fix(e2e): pause between clearState and launchApp so API 34 does not k…
eaff86a 
…ill the launch

On the first nightly (run 35173748419) every API 34 leg failed, on all three
RN versions, with a random flow stuck on a white screen waiting for
e2e-host-ready (#5104, #5105, #5106): 5 of 30 flows. API 35 and 36 lost none.

`launchApp: {clearState: true}` starts the app ~0.9 s after `pm clear`. The
clear removes the previous task, and when that removal's 1 s destroy timeout
fires Android 14 kills the package's current process - the one Maestro just
started:

  ActivityTaskManager: Destroy timeout of remove-task, attempt to kill Task{#10}
  ActivityManager: Killing 3179:com.rnvtest.host (adj -10000): remove task

Three such kills in the RN 0.82 leg's logcat, three failed flows. Reproduced on
a local API 34 emulator with `pm clear; sleep D; am force-stop; am start`:
2 of 15 launches killed at D=0.9, 0 of 15 at D=2.5.

Split the step and wait 2.5 s in between, Android only. Under Maestro the gap
from clear to process start goes from 1.3-1.8 s to 5.3-5.8 s locally; the cost
is about 4 s per flow on Android. This is a wait, not a retry (D11).
@mziolkowskii
fix(e2e): show Maestro's failure reason in the job summary
3132279 
Maestro 2.10 writes the reason as the text of the element
(`<failure>Assertion is false: id: evt-muted is visible</failure>`), while
junit-summary.mjs only read a `message` attribute. Every failed flow in the
first nightly (run 35173748419) therefore rendered with an empty Failure
column, and the reason was only available from the job log or the artifact.

Read the element text when there is no `message` attribute (entity-decoded,
CDATA verbatim). The attribute still wins when both are present. Counting
failures is unchanged, so history and nightly issues are unaffected.

Five new tests; three of them fail without the change.
@mziolkowskii
fix(e2e): answer a leftover "Open in app?" prompt before waiting for …
f9c7d56 
…the iOS host

On the nightly dispatched from this branch (run 35198102655) the RN 0.87 /
iOS 18 leg failed all 10 flows on `e2e-host-ready` (#5111). Every screenshot
shows the system "Open in RNVideoE2E?" prompt over a greyed-out app.

The warm-up flow exists to take that prompt, but on this runner (78 s first
launch) the prompt had not appeared 20 s after openLink, so the warm-up's
optional steps gave up and it surfaced afterwards. It belongs to SpringBoard:
it survives stopApp, clearState and launchApp, and while it is up the app is
not in the hierarchy. Nothing answered it, because the only "Open" handling
sits in open-scenario.yaml, after the wait that was failing. On the legs where
the warm-up did tap Open, everything passed.

launch-app.yaml now taps Open when the prompt is already visible after launch
(iOS only) and relaunches to drop whatever scenario the stale link opened.
Open rather than Cancel, so the approval is recorded.

Reproduced on a fresh iOS 26.5 simulator: leave the prompt unanswered after
`simctl openurl`, and the previous launch-app.yaml fails on e2e-host-ready
exactly as in CI; the new one taps Open, relaunches and passes, and a later
link opens without a prompt. With no prompt the check is skipped after ~7 s,
which overlaps with the app's start-up.
@mziolkowskii mziolkowskii changed the title fix(e2e): first nightly run fixes (iOS report path, API 34 launch race, failure reasons) fix(e2e): first nightly run fixes (iOS report path, API 34 launch race, leftover iOS prompt, failure reasons) Sep 17, 2026
@mziolkowskii
fix(test-app): build on iOS with RN 0.82 (fmt consteval and glog module)
7b0c666 
Both RN 0.82 iOS legs have failed to build on every nightly so far (#5109,
#5110): "could not build module 'fmt'" followed by "could not build
Objective-C module 'NitroModules'". Two independent errors sit behind that.

1. fmt. The existing workaround builds the fmt pod as C++17, which only
   covers the pod's own sources. On RN 0.82 the Swift compile of NitroModules
   0.37.1 builds the `fmt` clang module itself, with the consumer's C++20,
   and Xcode 26 rejects fmt 11.0.2's FMT_STRING consteval call sites again.
   fmt 11.0.2 has no override for FMT_USE_CONSTEVAL, so post_install turns
   it off in the installed base.h.

2. glog, visible only once fmt builds. NitroModules' umbrella reaches
   <glog/logging.h> (ReactProp.hpp -> React Fabric headers ->
   react_native_assert.h). logging.h includes log_severity.h and
   vlog_is_on.h inside `namespace google`; with the generated
   `module * { export * }` those are submodules, and clang refuses a module
   import inside a namespace. post_install declares the two as textual
   headers in glog.modulemap.

Both patches are idempotent and skipped when the file is absent (RN 0.87
links the prebuilt core). pod install always runs in CI, so they also apply
on top of a restored Pods cache.

Reproduced and verified locally on Xcode 26.6 with the 0.82 variant: the
NitroModules target fails with the CI errors before, and the whole
RNVideoE2E workspace builds after (BUILD SUCCEEDED).
@mziolkowskii
fix(e2e): open iOS deep links through the fixture server so a simctl …
d57da41 
…timeout is retried

On the second nightly from this branch (run 35204573155) the RN 0.77 / iOS 26
leg lost smoke-rate to the deep-link transport: `xcrun simctl openurl` gave up
after 13 s with NSPOSIXErrorDomain 60 on a runner where clearState alone took
15 s, and the flow failed - although the failure screenshot shows the app on
scenario:mp4, the link having arrived 10 s later.

open-scenario.yaml wrapped openLink in `retry` for exactly this, but the log
shows a single attempt followed by "Retry 2 times FAILED". Maestro 2.10's
retryCommand only catches MaestroException ("driver transport failures ...
propagate naturally"), and this failure is an IllegalStateException; `optional`
has the same limit. Nothing in a flow can recover from it.

So the transport moves out of the flow on iOS. The fixture server, which
already runs on the host for every leg, gets POST /__open-link: loopback only,
accepts nothing but rnvtest://scenario/<name>, runs simctl through execFile
and retries up to three times (open-link.mjs). Flows and the warm-up call it
from runScript; Android keeps Maestro's openLink. The dead `retry` is gone.
Still transport only: what the app shows is asserted by the flow and never
retried (D11).

Verified on an iOS 26.5 simulator with a fresh RN 0.82 build: warm-up and
three flows pass; with an xcrun wrapper that delivers the link and then
reports a timeout, every open takes two attempts, the link is delivered twice
and the flows still pass, since the test app keys its screen on the scenario.
@mziolkowskii
fix(e2e): address review of the nightly fixes
a2e8876 
- _e2e-ios.yml: restore the single quotes a formatter had flipped (14 lines
  of noise in the artifact fix), and clear maestro-tests before copying.
- The iOS workflow passes the resolved simulator to Maestro (-e SIM_UDID),
  the open-link script forwards it and the fixture server opens the link on
  exactly that device instead of `booted`. Validated as a UDID; absent means
  E2E_SIM_UDID, then `booted`, so local runs need nothing new.
- open-link-ios.js: SOFT=true reports a failed open instead of throwing. The
  warm-up uses it, so a failed open no longer stops it before it can tap
  Open. A server that is not running is reported as such, and an open that
  needed more than one attempt is logged to maestro.log.
- launch-app.yaml: the leftover-prompt check matches the prompt's title, not
  the bare "Open" button text.
- open-scenario.yaml says why Android has no retry: the removed `retry` never
  fired for a transport failure there either, and none has been seen.
- Podfile: pin the patched glog.modulemap's mtime. CocoaPods regenerates the
  file on every pod install, and a module map that looks modified would make
  clang rebuild glog and its importers on top of a restored DerivedData.
- junit-summary.mjs: a failure body keeps its first line, capped at 300
  characters (stack traces); several CDATA sections are joined; a ">" inside
  a quoted attribute no longer cuts the message short.

Checked on an iOS 26.5 simulator: stale prompt cleared via the title match;
warm-up continues with the server down, a real flow fails with a clear
message; explicit UDID reaches simctl. The Podfile hook was run against a
stub sandbox, including a regenerated module map. 188 unit tests pass.
@mziolkowskii
mziolkowskii force-pushed the ci/e2e-nightly-first-run-fixes branch from 8f42562 to a2e8876 Compare September 17, 2026 10:49
@mziolkowskii mziolkowskii changed the title fix(e2e): first nightly run fixes (iOS report path, API 34 launch race, leftover iOS prompt, failure reasons) fix(e2e): make the nightly truthful and green (report path, API 34 race, iOS prompt, RN 0.82 build, deep-link transport) Sep 17, 2026
@mziolkowskii mziolkowskii changed the title fix(e2e): make the nightly truthful and green (report path, API 34 race, iOS prompt, RN 0.82 build, deep-link transport) fix(e2e): make nightly results trustworthy and unblock iOS on RN 0.82 Sep 17, 2026
moskalakamil
moskalakamil requested changes Sep 17, 2026
View reviewed changes

@moskalakamil moskalakamil left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let's also clean up the comments- we have too many of them imo

Comment thread e2e/shared/open-link-ios.js Outdated
Comment thread e2e/fixtures/open-link.test.mjs
Comment thread e2e/fixtures/open-link.mjs
@mziolkowskii
fix(e2e): trim the comments and use const/let in the Maestro script
90f2a4d 
Review of #5113: the rationale and measurements behind the launch-app split, the iOS
prompt handling and the open-link retry stay in e2e/CONTEXT.md; the code comments now
say what each step guards against and point there. open-link-ios.js runs on GraalJS
(Maestro 2.x ships no Rhino), so ES5 var is not needed.
@moskalakamil
moskalakamil merged commit 410bede into master Sep 18, 2026
10 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@Okelm Okelm Okelm approved these changes
@moskalakamil moskalakamil moskalakamil approved these changes
@fnwk fnwk Awaiting requested review from fnwk

Assignees

No one assigned

Labels

None yet

Projects

React Native Video - Public Roadmap
Status: Done

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

[e2e nightly] e2e-android-rn0.77-api34-exclude-tags-flaky

3 participants

@mziolkowskii @Okelm @moskalakamil