Summary
test MQTT connections and commands > subscribe to retained record with patch operations (unitTests/apiTests/mqtt-test.mjs:437) is flaky under CPU contention: 3 failures in 30 local runs (~10%) of the full file with HARPER_STORAGE_ENGINE=lmdb, pinned to 2 cores (Linux, Node 26). Found while investigating the 2026-08-21 unit-flake cluster (epic #1655); it has not yet been observed red on CI main in that window, but at a 10% contended failure rate it is a red run waiting to happen.
Failure
Uncaught AssertionError [ERR_ASSERTION]: 2 == 3
at MqttClient.onMessage (unitTests/apiTests/mqtt-test.mjs:456:13)
assert.equal(messages[1].count, 3) sees count: 2 — i.e. the second message the subscriber receives is a duplicate of the PUT notification ({name: 'a starting point', count: 2}) rather than the PATCH notification (count: 3). Identical signature in all three failing runs.
Note it fails as an uncaught exception: the assertions run inside the client.on('message', ...) handler, outside the test's promise chain, so a failure kills the run as Uncaught AssertionError instead of failing the test cleanly — worth fixing along with the race (collect messages, assert after await).
Mechanism (hypothesis)
The test subscribes (QoS 1) to SimpleRecord/78, PUTs {count: 2}, waits 20ms, PATCHes to {count: 3}, and asserts messages arrive as exactly [put, patch] at messages.length === 2. The topic has retained-record semantics: a subscriber can legitimately receive both the retained/current-state delivery of a record AND the live update notification for the same version. When the PUT commits between subscription registration and the retained-state read, the client gets {count:2} twice (live + retained snapshot, order varies), so messages[1] is the duplicate. The pre-existing 20ms delay at mqtt-test.mjs:466-468 (added for "out of order on a loaded CI runner") band-aids the PUT/PATCH ordering but not the duplicate-delivery case.
The subscription-replay unit tests explicitly document this as expected retained-message behavior ("the subscriber may legitimately receive both — verify no key is lost", unitTests/resources/subscriptionReplay.test.js), so this looks like a test defect (over-strict exact-sequence assertion), not a product bug — the fix is to assert on the latest-value-per-version rather than exact message positions, or tolerate duplicate (id, version) deliveries.
Reproduction
HARPER_STORAGE_ENGINE=lmdb taskset -c 0,1 npx mocha unitTests/apiTests/mqtt-test.mjs
looped 30x against a CI-style fresh Harper install (DEFAULTS_MODE=dev ... harper.js install): failures on iterations 2, 3, 5.
Summary
test MQTT connections and commands > subscribe to retained record with patch operations(unitTests/apiTests/mqtt-test.mjs:437) is flaky under CPU contention: 3 failures in 30 local runs (~10%) of the full file withHARPER_STORAGE_ENGINE=lmdb, pinned to 2 cores (Linux, Node 26). Found while investigating the 2026-08-21 unit-flake cluster (epic #1655); it has not yet been observed red on CI main in that window, but at a 10% contended failure rate it is a red run waiting to happen.Failure
assert.equal(messages[1].count, 3)seescount: 2— i.e. the second message the subscriber receives is a duplicate of the PUT notification ({name: 'a starting point', count: 2}) rather than the PATCH notification (count: 3). Identical signature in all three failing runs.Note it fails as an uncaught exception: the assertions run inside the
client.on('message', ...)handler, outside the test's promise chain, so a failure kills the run asUncaught AssertionErrorinstead of failing the test cleanly — worth fixing along with the race (collect messages, assert afterawait).Mechanism (hypothesis)
The test subscribes (QoS 1) to
SimpleRecord/78, PUTs{count: 2}, waits 20ms, PATCHes to{count: 3}, and asserts messages arrive as exactly[put, patch]atmessages.length === 2. The topic has retained-record semantics: a subscriber can legitimately receive both the retained/current-state delivery of a record AND the live update notification for the same version. When the PUT commits between subscription registration and the retained-state read, the client gets{count:2}twice (live + retained snapshot, order varies), somessages[1]is the duplicate. The pre-existing 20ms delay at mqtt-test.mjs:466-468 (added for "out of order on a loaded CI runner") band-aids the PUT/PATCH ordering but not the duplicate-delivery case.The subscription-replay unit tests explicitly document this as expected retained-message behavior ("the subscriber may legitimately receive both — verify no key is lost", unitTests/resources/subscriptionReplay.test.js), so this looks like a test defect (over-strict exact-sequence assertion), not a product bug — the fix is to assert on the latest-value-per-version rather than exact message positions, or tolerate duplicate (id, version) deliveries.
Reproduction
looped 30x against a CI-style fresh Harper install (
DEFAULTS_MODE=dev ... harper.js install): failures on iterations 2, 3, 5.