AB#292773 test: make the SessionHelper tests deterministic - #146
AB#292773 test: make the SessionHelper tests deterministic#146eligutovsky wants to merge 4 commits into
Conversation
Both suites failed intermittently on CI while passing locally. Test-only changes; no production code is touched. AnalyticsHelperTests A trackEvent completion can outlive waitForExpectations, and tearDown then sets mockHttpClient and analyticsHelper to nil. Reading them back through self from inside a completion trapped on the implicit unwrap and took down the whole test binary, so the failure surfaced as an unrelated test in a later class. On CI the crash was reported against SessionHelperTests. Completions now capture what they need before tracking, holding their own strong reference. The two tests that used DispatchQueue.global().asyncAfter(deadline: .now() + 2) to order their phases now wait for the preceding flush to report completion. The sleep only made the ordering likely, and spent 2 of the 10 second budget doing nothing, which is what pushed the test over its timeout under load. MockKSHttpClientSingleFailure reports when the deliberately failed send has been delivered, so the retry test no longer guesses when the failure happened. Both mocks accumulated state on the flush queue while tests read it from completions, with no synchronization. That state is now behind an NSLock. SessionHelperTests Both tests encoded "does not block" as wall-clock measurements: an elapsed-time assertion of under 100ms, and a 2 second sleep inside a 3 second timeout. Scheduling delay alone can exceed those margins on a loaded runner while the behaviour is correct. They now assert ordering, using a semaphore the mock holds until the test releases it, so a timeout means a real hang rather than a slow machine. Suite runtime drops from 34s to 2.8s.
There was a problem hiding this comment.
Pull request overview
This PR aims to eliminate intermittent CI failures by making AnalyticsHelperTests and SessionHelperTests deterministic, removing reliance on wall-clock sleeps, and preventing late async completions from accessing torn-down test state.
Changes:
- Reworked
SessionHelperTeststo assert ordering (non-blocking behavior) via semaphores/expectations instead of elapsed-time assertions and sleeps. - Updated
AnalyticsHelperTeststo (a) capture strong references used in async completions to avoid post-tearDowncrashes, and (b) replaceasyncAftersequencing with completion-based sequencing. - Added basic synchronization (
NSLock) around shared mutable state in the HTTP client mocks used by the tests.
Reviewed changes
Copilot reviewed 2 out of 2 changed files in this pull request and generated 3 comments.
| File | Description |
|---|---|
| OptimoveSDK/Tests/Sources/Optimobile/SessionHelperTests.swift | Replaces timing-based “does not block” tests with ordering-based assertions using expectations/semaphores. |
| OptimoveSDK/Tests/Sources/Optimobile/AnalyticsHelperTests.swift | Makes analytics tests deterministic by avoiding sleeps, capturing needed references in async completions, and synchronizing mock state with locks. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
The suite stayed flaky after the previous commit. Root cause is that clearAnalyticsStore never cleared anything. AnalyticsHelper picks its store via getMainStoreUrl: with an app group defined it uses KAnalyticsDbShared.sqlite in the group container, otherwise KAnalyticsDb.sqlite under Documents. An app group IS defined in this test bundle, verified directly: isKumulosAppGroupDefined = true sharedDb exists = true .../AppGroup/.../KAnalyticsDbShared.sqlite appDb exists = false .../Documents/KAnalyticsDb.sqlite setUp only deleted the Documents path, a file that never exists, so every test in the class shared one accumulating store. Any event left behind by an earlier test is picked up by the next test's flush and delivered to that test's mock, and since the assertions are strict counts the expectation can never be satisfied, so the test burns its full 10 second timeout. Whether a preceding test leaves an event behind is itself a race, which is what made it intermittent: a failed send is not pruned, so it survives. clearAnalyticsStore now mirrors both branches of the production path selection. Confirmed with the leak deliberately introduced, one unpruned event before the other tests: old cleanup: 1 failure, 10s timeout, suite 11.169s new cleanup: 0 failures, suite 0.247s
- test_number_of_sent_events_from_background_threads_same_as_tracked was still racey: nothing ordered the background trackEvent calls before the final one, so if they had not been scheduled yet the final flush completed against a smaller store. The completion runs once, so it never re-checked and the test sat out its full timeout. Confirmed by forcing that interleaving with a 0.3s delay on the background blocks: the old test timed out, the new one passes. It now waits for the background events to be flushed first; they are still issued concurrently. - MockKSHttpClient.capturedData was documented as the last batch sent, but it returns every event delivered so far across all requests. Comment corrected. - Renamed test_sessionDidEnd_invokesTrackingCompletionAsynchronously: the mock is what defers, so the test does not verify asynchrony of the SDK's own call. That property is covered by test_sessionDidEnd_doesNotWaitForTrackingToComplete.
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 2 out of 2 changed files in this pull request and generated no new comments.
Suppressed comments (2)
OptimoveSDK/Tests/Sources/Optimobile/SessionHelperTests.swift:20
- In
blockingMock, the background block waits onreleaseTrackingwith no timeout. If the test fails before reachingreleaseTracking.signal()(e.g., an early assertion failure or the wait times out), that thread will remain blocked for the rest of the test run. Add a teardown cleanup that signals the semaphore to avoid leaking blocked threads across subsequent tests.
let releaseTracking = DispatchSemaphore(value: 0)
let trackingInvoked = expectation(description: "trackBackground was invoked")
let sessionDidEndReturned = expectation(description: "sessionDidEnd returned")
let blockingMock: (Date, @escaping SyncCompletedBlock) -> Void = { _, done in
trackingInvoked.fulfill()
DispatchQueue.global().async {
releaseTracking.wait()
done(nil)
}
OptimoveSDK/Tests/Sources/Optimobile/AnalyticsHelperTests.swift:176
- Typo in the test event type string: "immeditate_event" is likely meant to be "immediate_event". While it doesn’t affect the assertion, fixing it avoids confusion when debugging logs or failures.
analyticsHelper.trackEvent(eventType: "immeditate_event", properties: nil, immediateFlush: true)
#134 AB#292773 PR #134 fixes the same flake at the root: `storeUrlOverride` gives each test its own store, which is real isolation rather than deleting a database file that tests happen to share. That is strictly better than what this branch did, and keeping both would only conflict — the two touch the same setUp, tearDown and count assertions. AnalyticsHelperTests goes back to master's version here. The remaining timing work on those tests now lives in #150, on top of #134. SessionHelperTests is unaffected and stays: nothing else covers that sessionDidEnd must not block on tracking.
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 1 out of 1 changed files in this pull request and generated no new comments.
Suppressed comments (1)
OptimoveSDK/Tests/Sources/Optimobile/SessionHelperTests.swift:13
- PR description/title indicate changes to both AnalyticsHelperTests and SessionHelperTests to make tests deterministic, but in the current branch snapshot only SessionHelperTests reflects those determinism changes. For example,
OptimoveSDK/Tests/Sources/Optimobile/AnalyticsHelperTests.swiftstill contains wall-clock sleeps (asyncAfter(... + 2)) and itsclearAnalyticsStore()only deletes the Documents DB path, which contradicts the described fix for the app-group shared store. Please either include the missing AnalyticsHelperTests updates in this PR or adjust the PR title/description to match the actual changes.
// Both tests previously encoded "does not block" as wall-clock measurements:
// an elapsed-time assertion of under 100ms, and a 2 second sleep inside a 3
// second timeout. Neither survives a loaded CI runner, where scheduling delay
// alone can exceed the margin even though the behaviour is correct. They now
// assert ordering instead, so the timeouts are only a backstop against a real
// hang rather than the thing being measured.
AB#292773
Description of Changes
SessionHelperTestsasserted the wrong thing.sessionDidEnd()must return without waiting forthe background flush it kicks off — that is the whole point of the change in 6.2.6, "remove
semaphore wait in session end to avoid QoS inversion during background flush" — but the test only
checked that the tracking completion eventually ran. It would have passed just as happily if
sessionDidEnd()had gone back to blocking.The test now asserts the ordering directly. Tracking blocks on a semaphore the test releases only
after
sessionDidEnd()has returned, so a version that waited for tracking could not get past it:The old test is kept as
test_sessionDidEnd_invokesTrackingCompletion— renamed to say what itactually checks, since "does not wait" was never what it measured.
Scope was narrowed — the AnalyticsHelper half is gone
This PR originally also reworked
AnalyticsHelperTests. #134 fixes that flake better and thisbranch now stays out of its way.
Both of us independently found the same root cause: the simulator does not enforce entitlements, so
containerURL(forSecurityApplicationGroupIdentifier:)resolves for any group identifier andAnalyticsHelpersilently opens the shared-container database — while the test helper cleared theDocuments one, which was never the database in use. So the tests never had isolation and events
accumulated across the class.
Where the fixes differ:
storeUrlOverride: a fresh store per testPer-test stores are the right answer — the weakness I flagged in my own version was exactly that it
still rested on deleting a shared file. Keeping both would also have conflicted: they touch the same
setUp,tearDownand count assertions.AnalyticsHelperTestsis therefore back to master'sversion here, and the remaining timing work on it lives in #150, on top of #134.
Verification
Full suite: 75 tests, 0 failures.
Mutation-checked — restoring the semaphore wait in
SessionHelper.sessionDidEnd()failstest_sessionDidEnd_doesNotWaitForTrackingToCompleteand leavestest_sessionDidEnd_invokesTrackingCompletionpassing, which is the split the rename describes.Both tests run in 0.003s.
One thing this does not claim
I could not reproduce the CI flake locally, so I am not claiming a measured reduction anywhere.
Running the suite 8 times under full 11-core CPU load on unfixed
masterpassed 8/8 — the runslowed from 2.8s to 31s, so the load was real, but the flake needs something about the
GitHub-hosted runner. The argument here is about what the tests assert, not about flake rates.
Breaking Changes
Test-only. No version bump.
Release Checklist
pod lib lint— n/a, no shipped code changedIntegration tests
n/a — no shipped code changed.