Flaky CI Failure: Android Integration Tests — Run integration tests times out after 200/202 tests complete

[github-actions]

Flaky CI Failure: Android Integration Tests — Run integration tests times out after 200/202 tests complete

Workflow: Test
Failed run: https://github.com/xmtp/libxmtp/actions/runs/31221165776
Commit: 1b762b9
Failed jobs: test-android / Integration Tests (Android)

Summary

The Run integration tests step in the Android integration test job timed out after hitting the 20-minute step limit. The emulator completed 200 of 202 tests (with 3 skipped and 0 failed), but the final 2 tests hung indefinitely. Tests 195–200 showed significantly slower execution (~1.5 minutes each vs. ~10 seconds per test earlier in the run), suggesting the suite was slowing down toward the end before fully stalling.

Error Details

emulator-5560 - 14 Tests 194/202 completed. (3 skipped) (0 failed)    21:58:02
emulator-5560 - 14 Tests 195/202 completed. (3 skipped) (0 failed)    21:59:02  (+60s)
emulator-5560 - 14 Tests 196/202 completed. (3 skipped) (0 failed)    22:00:02  (+60s)
emulator-5560 - 14 Tests 197/202 completed. (3 skipped) (0 failed)    22:01:32  (+90s)
emulator-5560 - 14 Tests 198/202 completed. (3 skipped) (0 failed)    22:03:02  (+90s)
emulator-5560 - 14 Tests 199/202 completed. (3 skipped) (0 failed)    22:04:32  (+90s)
emulator-5560 - 14 Tests 200/202 completed. (3 skipped) (0 failed)    22:06:12  (+100s)
##[error]The action 'Run integration tests' has timed out after 20 minutes.  22:06:45

Last named skipped tests (around position 159–162):

org.xmtp.android.library.HistorySyncTest > testSyncDeviceArchive[emulator-5560 - 14] SKIPPED
org.xmtp.android.library.HistorySyncTest > testSyncMessages[emulator-5560 - 14] SKIPPED
org.xmtp.android.library.HistorySyncTlsTest > testArchiveRoundTripOverTls[emulator-5560 - 14] SKIPPED

Analysis

The step-level 20-minute timeout was hit with 2 tests still running, 0 tests explicitly failed. The progressive slowdown (from ~10s/test to ~90s/test near the end) indicates one or more tests were waiting on a stream, consent update, or async event that never arrived rather than crashing outright. The identity of the final 2 hanging tests is not visible in the log because no completion or failure line was emitted before the timeout. This is a test/code failure (likely a flaky streaming or sync test), not an infrastructure failure.

---

Reported by Flaky Failure Watcher

[github-actions]

Recurrence — 2026-08-07

Failed run: https://github.com/xmtp/libxmtp/actions/runs/31223948308
Commit: 3d28107

Same timeout pattern as the original report. The Run integration tests step hit the 20-minute limit with 201/202 tests completed (3 skipped, 0 failed). The last few tests slowed dramatically before stalling:

emulator-5560 - 14 Tests 194/202 completed. (3 skipped) (0 failed)    22:40:51
emulator-5560 - 14 Tests 195/202 completed. (3 skipped) (0 failed)    22:41:51  (+60s)
emulator-5560 - 14 Tests 196/202 completed. (3 skipped) (0 failed)    22:42:51  (+60s)
emulator-5560 - 14 Tests 197/202 completed. (3 skipped) (0 failed)    22:44:21  (+90s)
emulator-5560 - 14 Tests 198/202 completed. (3 skipped) (0 failed)    22:45:51  (+90s)
emulator-5560 - 14 Tests 199/202 completed. (3 skipped) (0 failed)    22:47:21  (+90s)
emulator-5560 - 14 Tests 200/202 completed. (3 skipped) (0 failed)    22:49:01  (+100s)
emulator-5560 - 14 Tests 201/202 completed. (3 skipped) (0 failed)    22:50:31  (+90s)
##[error]The action 'Run integration tests' has timed out after 20 minutes.  22:51:12

This run got one further (201/202 vs 200/202 previously) before timing out, but the same progressive slowdown pattern is present. One test remained stuck indefinitely.

[insipx]

Data point from a different cause that produced the identical signature, in case it helps narrow the mechanism.

On #3954 I hit this exact shape — 200/202 completed, 0 failed, then a step timeout — and there it was fully explained by extra per-client network I/O. That PR (transiently) had a 3-second liveness-check throttle, so every client build more than 3s after registration did an extra fetch_key_packages round trip. The tell was in the progress lines: the run is brisk to ~194/202 (10-20 tests per 10s tick), then the last handful crawl to roughly one test per 90s before the step is killed. Removing the extra per-build network call fixed it, and that job is green again.

I'm explicitly not claiming that is this issue's root cause — this one reproduces on unrelated branches, so something else is driving it. But the two share a mechanism worth checking: the tail crawl is not uniform slowness, it is the last few tests degrading sharply, which looks much more like a resource that accumulates per-test and saturates near the end (sockets/connections, emulator-side network state, or a backend the emulator talks to getting slower under accumulated load) than like any single slow test. Two concrete suggestions for whoever picks this up:

1. The per-10s progress lines are a usable profile. Diffing tick deltas between a green and a red run should show whether the knee is always at the same test index (points at a specific test) or at a wall-clock/resource threshold (points at accumulation).
2. Because it fails with 0 failed, no test is actually asserting anything wrong — the budget is being consumed elsewhere. Worth confirming whether the emulator or the shared backend is the saturating side before touching test code.

Happy to be pinged if the accumulation theory pans out, but I'm not picking this up.

[insipx]

Correcting my comment above — I was wrong, please disregard the causal claim.

I said my case was "fully explained by extra per-client network I/O" and that removing it made the job green. Both halves are false, and I should not have asserted them before re-running.

I removed the extra per-build fetch_key_packages on #3954 and the Android job failed again with the identical signature: brisk to 194/202, then a tail crawl, 0 failed, step timeout. It reached 201/202 instead of 200/202, which is noise, not a fix. My change neither caused this nor fixed it. Independently confirmed it is pre-existing: test-android / Integration Tests (Android) failed on main in 4 of the last 5 failing runs, including runs 31223948308 and 31221165776 from earlier today, before my branch existed.

What I do think is still worth keeping from that comment is one observation, now stated as an observation rather than a cause. The tail crawl has an exact 90-second cadence. From the run on my branch:

23:58:15  194/202
23:59:15  195/202   (+60s)
00:00:35  196/202   (+90s)
00:02:05  197/202   (+90s)
00:03:35  198/202   (+90s)
00:05:05  199/202   (+90s)
00:06:35  200/202   (+90s)
00:08:05  201/202   (+90s)

Six consecutive intervals at exactly 90s, to the second. That regularity argues against the resource-saturation theory I floated earlier — saturation would degrade raggedly. A precise, repeating 90s per test looks like a fixed timeout being hit and then something proceeding: a client/network operation with a 90s deadline, a retry ceiling, or a poll interval, in whichever tests happen to run last in this shard's ordering. Someone who knows the Android harness could likely map 90s onto a specific configured timeout quickly.

Apologies for the noise on the earlier comment.

[github-actions]

Recurrence — 2026-08-20

Failed run: https://github.com/xmtp/libxmtp/actions/runs/32408859442
Commit: c30ac5f

Same timeout pattern as before, but this time 201/202 tests completed (vs 200/202 in the original report) before the 20-minute step limit was hit.

emulator-5560 - 14 Tests 194/202 completed. (3 skipped) (0 failed)
emulator-5560 - 14 Tests 195/202 completed. (3 skipped) (0 failed)
emulator-5560 - 14 Tests 196/202 completed. (3 skipped) (0 failed)
emulator-5560 - 14 Tests 197/202 completed. (3 skipped) (0 failed)
emulator-5560 - 14 Tests 198/202 completed. (3 skipped) (0 failed)
emulator-5560 - 14 Tests 199/202 completed. (3 skipped) (0 failed)
emulator-5560 - 14 Tests 200/202 completed. (3 skipped) (0 failed)
emulator-5560 - 14 Tests 201/202 completed. (3 skipped) (0 failed)
##[error]The action 'Run integration tests' has timed out after 20 minutes.

The slowdown pattern at the end of the suite persists — the last few tests each take 1–2 minutes, then the final test hangs indefinitely. The 0 failed count indicates no explicit test assertion failures, just a hang.