Flaky unit test: MQTT 'subscribe with QoS=1 and reconnect with non-clean session' silent 20s timeout (lmdb pass)

[kriszyp]

Summary

test MQTT connections and commands > subscribe with QoS=1 and reconnect with non-clean session (unitTests/apiTests/mqtt-test.mjs) timed out at its 20s mocha limit on main, taking the Unit Test workflow red at head-of-main. Filed to track the flake behind its quarantine (the test is skipped with a pointer here until this is root-caused).

Evidence run: Unit Test run 32531526557, job "Unit Test (Node.js v26)" on main (sha f8a5aa9), 2026-08-21.

What the CI log shows

• The failure happened in the lmdb pass (test:unit:lmdb → HARPER_STORAGE_ENGINE=lmdb npm run test:unit:apitests), not the default rocksdb pass — the same test had passed the rocksdb pass minutes earlier in the same job (62ms).
• The test's beforeEach ran (its four clear() is deprecated lines appear at 22:12:28.26), then nothing for 20.15s until mocha's Error: Timeout of 20000ms exceeded at 22:12:48.41. No [mqtt] server-side log lines in between.
• The failure is a silent hang, not an assertion: the test's own final wait (the 3-queued-messages check) has a 15s bounded reject with a descriptive message that never fired, so the hang was in one of the earlier steps that have no individual timeout: one of the three connectAsync calls, subscribeAsync, one of the three endDurableSession waits (each awaits a server-side 'disconnected' event), the QoS-1 publish/acknowledge wait (awaits a server-side 'acknowledged' event), or one of the three publishAsync (PUBACK) waits.

Local reproduction attempt

Not reproduced: 30 consecutive full runs of mqtt-test.mjs under HARPER_STORAGE_ENGINE=lmdb, pinned to 2 cores (with unrelated suites loading the other cores) on Linux/Node 26 — 0 failures. Single-test runs also pass consistently.

Hypotheses worth checking when this is picked up

1. Reconnect racing prior session teardown. The test repeatedly does endDurableSession (which resolves on the server's 'disconnected' event) and immediately reconnects with the same clientId (test-client1, clean: false). If 'disconnected' is emitted before the durable session's async teardown/persistence completes, the next CONNECT for the same clientId races the takeover and could stall before CONNACK.
2. QoS downgrade on delivery. The publish/acknowledge step resolves only when the server emits 'acknowledged' for the client's PUBACK. If the delivery path ever hands the message over at QoS 0 (e.g. via the current-state/queued-delivery path rather than the live listener), no PUBACK ever arrives and the await hangs silently.
3. lmdb-specific timing in durable-session persistence (the failure occurred only in the lmdb pass so far — one observation, so weak evidence).

Suggested reinstatement path

Wrap each unbounded await in the test with a labeled bounded timeout (per-step AbortSignal.timeout/Promise.race with a descriptive error) so the next occurrence pinpoints the hanging step, then unskip and let CI collect evidence.