Skip to content

fix(e2e): harden nightly legs against transport-level failures - #5129

Merged
fnwk merged 4 commits into
masterfrom
fix/e2e-nightly-flakes
Sep 22, 2026
Merged

fnwk merged 4 commits into
masterfrom
fix/e2e-nightly-flakes

Conversation

@mziolkowskii

@mziolkowskii mziolkowskii commented Sep 22, 2026 •

Copy link
Copy Markdown
Collaborator

Summary

The first four nightly E2E runs from master (2026-09-19 → 09-22) each had one or two red legs, for four unrelated reasons, none of them in the library. This PR removes three of those failure modes at the layer they live in, without retrying anything the flows assert, and documents the fourth.

Night Red leg Cause Here
09-19 android 0.82 / API 35 android-settle.sh: uiautomator dump returned 0 without writing the file; the adb shell cat under set -e killed the script before Maestro started fixed
09-20 android 0.82 / API 35 Gradle: 403 from Maven Central for gson-2.9.1.jar while resolving the build classpath fixed
09-21, 09-22 ios 0.87 / iPhone 17 (iOS 26) smoke-mute-volume: the btn-mute tap never reached JS, so evt-muted never appeared transport retry, root cause open (#5130)
09-22 android 0.87 / API 36 DeviceServerDiedException ... UNAVAILABLE on the first command of the first flow, 3 s after the driver came up; the other nine flows passed documented only

Motivation

Nightly is meant to become a required check after 10–15 consecutive clean runs. Each of these failures would reset that streak while telling us nothing about the library.

For the iOS one, the simulator device.log from both failing nights shows UIKit delivering touch-down and touch-up to the app's window, followed by no onPress. In both failures the touch-down was followed by ~120 ms without touch updates, then a UIKit system-gesture state change, then the touch-up; in every passing run of that flow the state change came after the touch-up. The tap went 2.4 s after onEnded, so the settle added in wait-for-end.yaml after the 09-15 incident is not a complete answer. Evidence and open questions are in #5130.

The Android driver death gets no mitigation here. A warm-up maestro test before the suite (tried in an earlier revision of this PR) does not help: every maestro test process installs, connects and uninstalls its own driver, so the suite's first connection is still a first connection. A transport-level retry of flows that die in the driver is a possible follow-up.

Changes

  • Test app: eventLog.press() counts presses per control and EventLogPanel renders one pressed-<testID>-<n> marker per press in the fixed marker row, above the scrolling log, so a flow can wait for a specific tap on an element that is never off-screen. Unit tests added.
  • e2e/shared/press.yaml (new): taps a control and waits for that marker, re-sending the tap (max 2) only if it never appeared. The marker is set synchronously in onPress before the player is touched, so the retry covers tap transport only; every player marker is still asserted exactly once, and the controls set absolute values, so a second delivery is harmless. Used by the five flows that press on an idle player (smoke-mute-volume, smoke-seek, smoke-hls-seek, smoke-loop, smoke-replay-after-end); taps issued during playback stay bare tapOn, since iOS delivers those only once the player is idle.
  • scripts/e2e/android-settle.sh: one guest shell per attempt; the dump file, not the exit status, decides whether an attempt produced a dump; only a dialog still on screen in the last dump fails the leg. New android-settle.test.mjs (fake adb/sleep on PATH), 6 cases including "dialog dismissed, then no dumps".
  • .github/workflows/_e2e-android.yml: assembleDebug gets one more attempt after 30 s only when its log shows a 403/429/5xx from a repository; a compile error or a 404 still fails once.
  • Docs: e2e/CONTEXT.md (lost-tap gotcha linking E2E: iOS 26 tap on an idle player reaches the app window but never fires onPress #5130, the second transport-only retry, why a driver warm-up does not work), e2e/README.md.

Platforms affected

  • Android
  • iOS
  • visionOS
  • tvOS / Android TV
  • Windows
  • Web

Type of change

  • Bug fix (non-breaking change that fixes an issue)
  • New feature (non-breaking change that adds functionality)
  • Breaking change (fix or feature that changes existing behavior)
  • Documentation only

Test plan

  • Automated: test-app/src/e2e/eventLog.test.ts (press counting, snapshot identity, reset), scripts/e2e/android-settle.test.mjs (6 cases). bun run test 197 pass, bun lint and bun typecheck clean.
  • Flows: no assertion changed. Each flow asserts the same player markers once; only the tap transport is wrapped, and only for taps on an idle player.
  • CI: the PR gate runs the Android legs with the new settle and build steps and the iOS leg with the press subflow on five flows. The nightly matrix is the real test for the iOS change (the failure showed 2 nights out of 4 on one leg); a re-sent tap shows up as a retry attempt in the flow's commands.json in the Maestro debug artifact, so the rate of lost taps stays observable.
  • Not reproduced locally: the iOS tap loss needs a hosted iOS 26 simulator under load; evidence is from the nightly artifacts (runs 35553773836 and 35678849766).

Checklist

  • I read the contributing guidelines.
  • I added or updated tests that cover this change, or explained under "Test plan" why none apply.
  • For a bug fix the E2E suite can exercise, this PR includes a Maestro flow that fails without the fix, or "Test plan" explains why none does.
  • bun run test, bun lint and bun typecheck pass locally.
  • I updated the documentation / README where relevant.
  • If this changes how the library is used (API, props, events, behavior), I updated the AI agent skill (skills/react-native-video/) to match. (Not applicable: no library change.)
  • This PR is focused on a single concern.
  • I tested my changes on at least one platform, or described under "Test plan" what CI (which runs both) should show.

Nightly 2026-09-21 and 09-22 (iOS 26, RN 0.87) failed smoke-mute-volume at
`evt-muted`: Maestro reported the btn-mute tap done 2.4 s after onEnded, the
simulator log shows touch-down and touch-up delivered to the app's window, and
onPress never ran, so `muted = true` was never called. Same symptom as the tap
lost 264 ms after onEnded on 09-15, so the settle in wait-for-end.yaml is not a
complete answer. The other iOS legs were 12/12 green those nights. Root cause
not established: #5130.

The test app now counts presses per control and renders one marker per press,
`pressed-<testID>-<n>`, in the fixed marker row above the scrolling log, so a
flow can prove a specific tap arrived on an element that is never off-screen.
e2e/shared/press.yaml taps a control and waits for that marker, re-sending the
tap (retry, 2 max) only when it never appeared. The marker is set synchronously
in onPress before the player is touched, so the retry covers tap transport and
nothing else; every player marker is still asserted once, and every control
sets an absolute value, so a second delivery is harmless. The five flows that
press on an idle player (after wait-for-end.yaml) use it; taps issued during
playback stay bare, because iOS delivers those only once the player is idle,
later than the subflow waits.
… no file

Nightly 2026-09-19 (RN 0.82, API 35) failed before Maestro started: 3 s after
install `uiautomator dump` returned 0 without writing /sdcard/settle.xml (the
guest was still busy; adb had reported "device offline" moments earlier), and
the unguarded `adb shell cat` under `set -e` killed the script. The retry loop
assumed only the dump command itself could fail.

Each attempt is now one guest shell (rm, dump, cat) and the file, not the exit
status, decides whether it produced a dump; a missing one is retried. Only a
dialog still on screen in the last dump fails the leg: no dump at all, or a
dismissed dialog followed by dump-less attempts, ends in a warning and exit 0,
because there is nothing to fail on before a single flow has run and Maestro's
own driver start-up is the next check of the guest.

android-settle.test.mjs runs the script against a fake `adb` and `sleep` on
PATH (bun test), including the dump-less and dismissed-then-dump-less cases.
@mziolkowskii mziolkowskii changed the title fix(e2e): harden nightly legs against four transport-level failures fix(e2e): harden nightly legs against transport-level failures Sep 22, 2026
…rror

Nightly 2026-09-20 (RN 0.82, API 35): Gradle could not download gson-2.9.1.jar
for the build classpath, Maven Central answered 403, and the build failed
after 8 s. Gradle retries 5xx and timeouts itself, not 403. `assembleDebug`
gets one more attempt after 30 s, only when its log shows a 403, 429 or 5xx
from a repository; a compile error or a missing artifact (404) still fails
once.
@mziolkowskii
mziolkowskii requested a review from fnwk September 22, 2026 07:23
@fnwk
fnwk merged commit 6dd359f into master Sep 22, 2026
10 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

Development

Successfully merging this pull request may close these issues.

2 participants