Repository navigation
fix(e2e): make nightly results trustworthy and unblock iOS on RN 0.82 #5113
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
+471
−22
Merged
Changes from all commits
Commits
Show all changes
9 commits
Select commit
Hold shift + click to select a range
e91d1b9
ci(e2e): keep the iOS artifact rooted at the workspace so nightly fin…
mziolkowskii eaff86a
fix(e2e): pause between clearState and launchApp so API 34 does not k…
mziolkowskii 3132279
fix(e2e): show Maestro's failure reason in the job summary
mziolkowskii f9c7d56
fix(e2e): answer a leftover "Open in app?" prompt before waiting for …
mziolkowskii 7b0c666
fix(test-app): build on iOS with RN 0.82 (fmt consteval and glog module)
mziolkowskii d57da41
fix(e2e): open iOS deep links through the fixture server so a simctl …
mziolkowskii a2e8876
fix(e2e): address review of the nightly fixes
mziolkowskii 90f2a4d
fix(e2e): trim the comments and use const/let in the Maestro script
mziolkowskii 0857df2
Merge branch 'master' into ci/e2e-nightly-first-run-fixes
mziolkowskii File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,53 @@ | ||
| // Opens a scenario deep link on the iOS simulator for a Maestro flow (POST /__open-link, | ||
| // called from e2e/shared/open-link-ios.js), retrying `simctl openurl` when it times out. | ||
| // | ||
| // Why not Maestro's own `openLink`: on a hosted macOS runner `simctl openurl` has timed | ||
| // out (NSPOSIXErrorDomain 60) while the link still arrived seconds later, and Maestro's | ||
| // `retry`/`optional` do not catch that failure (it is not a MaestroException). Only the | ||
| // transport is retried; what the app then shows is asserted by the flow, never retried | ||
| // (the zero-retries rule in e2e/CONTEXT.md). | ||
| import { execFile } from 'node:child_process'; | ||
| import { promisify } from 'node:util'; | ||
|
|
||
| const execFileAsync = promisify(execFile); | ||
|
|
||
| // Deliberately narrower than the app's parser: this value becomes a process argument. | ||
| const SCENARIO_LINK = /^rnvtest:\/\/scenario\/[a-z0-9]+(?:-[a-z0-9]+)*$/; | ||
|
|
||
| const SIMULATOR_UDID = /^[0-9A-F]{8}(?:-[0-9A-F]{4}){3}-[0-9A-F]{12}$/i; | ||
|
|
||
| // The link from `?url=<link>`, or null unless it is exactly one scenario link. | ||
| export function scenarioLink(requestUrl) { | ||
| const values = new URL(requestUrl, 'http://x').searchParams.getAll('url'); | ||
| return values.length === 1 && SCENARIO_LINK.test(values[0]) ? values[0] : null; | ||
| } | ||
|
|
||
| // The simulator from `&device=<udid>`: undefined when absent, null when not a UDID. | ||
| export function simulatorDevice(requestUrl) { | ||
| const values = new URL(requestUrl, 'http://x').searchParams.getAll('device'); | ||
| if (values.length === 0 || (values.length === 1 && values[0] === '')) return undefined; | ||
| return values.length === 1 && SIMULATOR_UDID.test(values[0]) ? values[0] : null; | ||
| } | ||
|
|
||
| export async function openLink({ | ||
| link, | ||
| device = 'booted', | ||
| xcrun = 'xcrun', | ||
| attempts = 3, | ||
| pauseMs = 2000, | ||
| // simctl's own timeout is ~13 s; this only bounds a hung process. | ||
| run = (file, args) => execFileAsync(file, args, { timeout: 60_000 }), | ||
| sleep = (ms) => new Promise((resolve) => setTimeout(resolve, ms)), | ||
| }) { | ||
| let error = ''; | ||
| for (let attempt = 1; attempt <= attempts; attempt++) { | ||
| try { | ||
| await run(xcrun, ['simctl', 'openurl', device, link]); | ||
| return { ok: true, attempts: attempt }; | ||
| } catch (err) { | ||
| error = err.message; | ||
| if (attempt < attempts) await sleep(pauseMs); | ||
| } | ||
| } | ||
| return { ok: false, attempts, error }; | ||
| } |
|
mziolkowskii marked this conversation as resolved.
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,109 @@ | ||
| import { describe, test, expect } from 'bun:test'; | ||
| import { scenarioLink, simulatorDevice, openLink } from './open-link.mjs'; | ||
|
|
||
| const request = (link) => `/__open-link?url=${encodeURIComponent(link)}`; | ||
|
|
||
| describe('scenarioLink', () => { | ||
| test('accepts a scenario deep link', () => { | ||
| expect(scenarioLink(request('rnvtest://scenario/mp4'))).toBe('rnvtest://scenario/mp4'); | ||
| expect(scenarioLink(request('rnvtest://scenario/broken-manifest'))).toBe('rnvtest://scenario/broken-manifest'); | ||
| expect(scenarioLink(request('rnvtest://scenario/error-404'))).toBe('rnvtest://scenario/error-404'); | ||
| }); | ||
|
|
||
| test('rejects anything that is not exactly a scenario deep link', () => { | ||
| for (const link of [ | ||
| 'https://example.com', | ||
| 'rnvtest://other/mp4', | ||
| 'rnvtest://scenario/', | ||
| 'rnvtest://scenario/mp4/extra', | ||
| 'rnvtest://scenario/mp4?x=1', | ||
| 'rnvtest://scenario/MP4', | ||
| 'rnvtest://scenario/mp4 --help', | ||
| 'rnvtest://scenario/mp4;rm', | ||
| ' rnvtest://scenario/mp4', | ||
| ]) { | ||
| expect(scenarioLink(request(link))).toBeNull(); | ||
| } | ||
| }); | ||
|
|
||
| test('rejects a missing or repeated url parameter', () => { | ||
| expect(scenarioLink('/__open-link')).toBeNull(); | ||
| expect(scenarioLink('/__open-link?url=')).toBeNull(); | ||
| expect( | ||
| scenarioLink('/__open-link?url=rnvtest%3A%2F%2Fscenario%2Fmp4&url=rnvtest%3A%2F%2Fscenario%2Fhls') | ||
| ).toBeNull(); | ||
| }); | ||
| }); | ||
|
|
||
| describe('simulatorDevice', () => { | ||
| const udid = '7D16E67F-1E34-43CB-BD36-B95CCE24981A'; | ||
|
|
||
| test('is undefined when the request names no device', () => { | ||
| expect(simulatorDevice('/__open-link?url=x')).toBeUndefined(); | ||
| expect(simulatorDevice('/__open-link?url=x&device=')).toBeUndefined(); | ||
| }); | ||
|
|
||
| test('accepts a simulator UDID', () => { | ||
| expect(simulatorDevice(`/__open-link?url=x&device=${udid}`)).toBe(udid); | ||
| expect(simulatorDevice(`/__open-link?url=x&device=${udid.toLowerCase()}`)).toBe(udid.toLowerCase()); | ||
| }); | ||
|
|
||
| test('is null for anything else, including simctl keywords and repeats', () => { | ||
| for (const device of ['booted', 'all', '--help', `${udid} x`, 'iPhone%2017']) { | ||
| expect(simulatorDevice(`/__open-link?url=x&device=${device}`)).toBeNull(); | ||
| } | ||
| expect(simulatorDevice(`/__open-link?url=x&device=${udid}&device=${udid}`)).toBeNull(); | ||
| }); | ||
| }); | ||
|
|
||
| describe('openLink', () => { | ||
| const link = 'rnvtest://scenario/mp4'; | ||
| const timedOut = Object.assign(new Error('Simulator device failed to open'), { code: 60 }); | ||
|
|
||
| // `run` fails `failures` times, then succeeds. | ||
| function harness(failures) { | ||
| const calls = []; | ||
| const pauses = []; | ||
| return { | ||
| calls, | ||
| pauses, | ||
| run: async (file, args) => { | ||
| calls.push([file, ...args]); | ||
| if (calls.length <= failures) throw timedOut; | ||
| }, | ||
| sleep: async (ms) => void pauses.push(ms), | ||
| }; | ||
| } | ||
|
|
||
| test('runs `simctl openurl` once when it succeeds', async () => { | ||
| const h = harness(0); | ||
| expect(await openLink({ link, run: h.run, sleep: h.sleep })).toEqual({ ok: true, attempts: 1 }); | ||
| expect(h.calls).toEqual([['xcrun', 'simctl', 'openurl', 'booted', link]]); | ||
| expect(h.pauses).toEqual([]); | ||
| }); | ||
|
|
||
| test('targets the given simulator and xcrun binary', async () => { | ||
| const h = harness(0); | ||
| await openLink({ link, run: h.run, sleep: h.sleep, device: 'ABCD-1234', xcrun: '/tmp/fake-xcrun' }); | ||
| expect(h.calls).toEqual([['/tmp/fake-xcrun', 'simctl', 'openurl', 'ABCD-1234', link]]); | ||
| }); | ||
|
|
||
| test('tries again after a failure, pausing in between', async () => { | ||
| const h = harness(2); | ||
| expect(await openLink({ link, run: h.run, sleep: h.sleep, pauseMs: 250 })).toEqual({ ok: true, attempts: 3 }); | ||
| expect(h.calls).toHaveLength(3); | ||
| expect(h.pauses).toEqual([250, 250]); | ||
| }); | ||
|
|
||
| test('gives up after `attempts` failures and reports the last error', async () => { | ||
| const h = harness(Infinity); | ||
| expect(await openLink({ link, run: h.run, sleep: h.sleep, attempts: 3 })).toEqual({ | ||
| ok: false, | ||
| attempts: 3, | ||
| error: 'Simulator device failed to open', | ||
| }); | ||
| expect(h.calls).toHaveLength(3); | ||
| // No pause after the last attempt: nothing follows it. | ||
| expect(h.pauses).toHaveLength(2); | ||
| }); | ||
| }); |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.