Skip to content

Replace the Maestro flow with gekko UI tests for Android and iOS - #140

Open
rschattauer wants to merge 1 commit into
joreilly:mainfrom
rschattauer:main
Open

rschattauer wants to merge 1 commit into
joreilly:mainfrom
rschattauer:main

Conversation

@rschattauer

Copy link
Copy Markdown

Summary

Replaces maestro/GalwayBus.flow with a gekko test that runs
unchanged on Android and iOS, and switches the two CI jobs over. The scenario is the same as before: fresh
install -> Routes tab -> route 402 -> map with live bus positions.

Locally the UI test runs about 4-6x faster than the Maestro flow (table below).

What is gekko?

gekko runs UI tests written once in YAML on Android and iOS, just faster and with the possibility to self-heal.

  • In-app agent. A small agent inside the debug build reads the UI tree directly and acts through accessibility
    actions, so steps take milliseconds and there are no screenshot heuristics. It knows when the screen has settled,
    so there are no hard-coded sleeps.
  • System runner (UiAutomator / XCUITest) for what lives outside the app, e.g. permission dialogs.
  • Native Kotlin CLI (gekko run). On iOS the CLI injects the agent into the simulator app at launch, so the
    iOS project needs no changes and no dependency.
  • No model decides pass/fail. CI replays the YAML deterministically. An LLM-based --heal mode exists but is
    opt-in and is not used here. This could be used locally to fix/create/edit UI tests!

Why

  • Speed. Shorter feedback on the UI-test jobs (numbers below).
  • Stable selectors. The Maestro flow matched English strings from a live backend ("Seacrest - Eyre Square -
    Merlin Park", "Route 402", "Updated.*") and needed waitForAnimationToEnd workarounds. The gekko test selects
    by ids shared by both platforms and asserts on structure, not on live content.
  • One test, two platforms. Differences are expressed with platform: (the Android-only "Map loaded" check).
  • Disclosure: gekko is an open-source tool from Jakala, where I work.

Results

Whole-command wall-clock seconds, from cleared app data. 1 untimed warm-up, then 10 timed runs per cell; all runs
passed.

Platform Tool Mean Median Min-max Speed-up (mean)
Android gekko 1.1.0 2.8 s 2.8 s 2.5-3.0 5.6x
Android Maestro 2.11.0 15.3 s 15.0 s 14.3-17.8
iOS gekko 1.1.0 4.6 s 4.6 s 4.4-5.0 3.6x
iOS Maestro 2.11.0 16.8 s 16.6 s 15.7-18.1

Method: gekko ran against this branch's build (with the agent), Maestro against the previous upstream build with
the original flow; the app was reinstalled from the matching build before each block. Headless Google APIs
emulator (API 36) and an iPhone simulator on one Mac. The times include tool start-up, but not app build,
emulator boot or install. On GitHub-hosted runners those dominate the job, so the job-level saving will be smaller
than the test-level one.

Things to know

  • Only the debug variant gets the agent.
  • The new testTags are behaviour-neutral. The Route.Map.Loaded step needs a working Maps key, which CI already
    provides through GOOGLE_API_KEY; without one the map never reports "loaded".
  • The CI job maestro is renamed ui-tests. Update the required status check if branch protection uses it.
  • The test depends on route 402 existing in the live feed, as the Maestro flow did.

Testing

  • gekko run passes on Android (emulator) and iOS (simulator), 10+ runs each
  • Workflows linted with actionlint
  • Android "Map loaded" step verified with the flag forced, because I have no Maps key locally
  • The updated workflows have not run on GitHub-hosted runners yet; this PR's checks will be the first run

Run locally: ./gradlew :androidApp:assembleDebug, then
gekko run --project gekko --platform android --install (--platform ios for the simulator build).

Happy to adjust, or to keep the Maestro flow alongside for a while if you prefer.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant