Skip to content

test(utils): report SleeperThread watchdog expiry as its own outcome - #5432

Open
twstagg wants to merge 2 commits into
Alexays:masterfrom
twstagg:fix/sleeper-thread-watchdog
Open

twstagg wants to merge 2 commits into
Alexays:masterfrom
twstagg:fix/sleeper-thread-watchdog

Conversation

@twstagg

@twstagg twstagg commented Nov 2, 2026

Copy link
Copy Markdown

Symptom

Once configure succeeds (see #5431), the utils suite is what fails in a loaded build:

test/utils/sleeper_thread.cpp:79: FAILED:
  REQUIRE( run_in_subprocess(run_control_flag_stress) == 0 )
with expansion:
  -1 == 0

assertions: 756 | 755 passed | 1 failed
3/3 waybar:utils    FAIL    5.52s   exit status 42

It looks like a SleeperThread defect. It is not: it is the test's own watchdog, and it only fires when the machine is busy, so it reads as a flake.

Cause

run_in_subprocess starts the task under alarm(5) and leaves SIGALRM at its default disposition, which terminates the child. The parent then sees !WIFEXITED(status) and returns -1 — the same value it returns for a fork failure or a crash. So a child that simply needed more than five seconds is reported identically to a genuine logic failure, and the message gives no hint that a timeout happened at all.

The stress loop is wall-clock hungry rather than slow: it runs 200 iterations of thread creation, 100 wake_up() calls, stop() and join(), so under scheduling pressure every iteration pays for context switches. Measured on a 16-core machine saturated with 16 busy loops, the loop still completed, taking up to 30.8 s:

tree runs under load failures slowest run
master 40 19 5063 ms
master + #5431 40 25 5059 ms
this change 40 0 30775 ms

Every failure sits at ~5.0 s, i.e. exactly at the alarm. The five-second budget is therefore an undeclared performance assertion attached to a correctness test, and it fails on shared CI runners and on any machine doing parallel builds — the Nix build in particular.

Change

  • Give the watchdog a generous budget (60 s, comfortably above the 30.8 s worst case measured under full-core saturation) so it guards against a hang, which is its actual purpose, instead of also asserting speed.
  • Install a handler that exits with a dedicated code (124) on expiry, so an expiry is distinguishable from a crash instead of arriving as "killed by a signal".
  • Report the outcome in the failure message, so the next person sees watchdog expired: the stress loop did not finish… rather than -1 == 0.

The stress magnitude is deliberately unchanged — the volume of concurrent wake and stop calls is the point of the test.

Verification

  • utils suite: 772 assertions in 44 test cases, all passing.
  • 40 runs of [sleeper_thread] under 16-core saturation: 0 failures (was 19–25 of 40).
  • Full Nix build with tests enabled: Ok: 3, Fail: 0.

Depends on #5431 only in the sense that it is what you hit next; the two changes are independent.

The subprocess watchdog was alarm(5) with the default SIGALRM action, so a
child that needed longer than five seconds died from the signal and the parent
returned -1, indistinguishable from a crash or a failed assertion. The stress
loop is wall-clock heavy rather than slow: on a machine saturated with 16 busy
loops it was measured taking up to 30.8s while still completing, and 19 of 40
runs failed at the five second mark. That makes the test a disguised
performance assertion, and it fails in loaded CI builders.

Give the watchdog a generous budget and install a handler that exits with a
dedicated code, so an expiry is reported as an expiry instead of -1, and add
the outcome to the failure message. The stress magnitude is unchanged: the
point of the test is the volume of concurrent wake and stop calls.

Measured after the change: 0 of 40 runs fail under the same load, and the full
utils suite passes with 772 assertions in 44 cases.
@twstagg
twstagg force-pushed the fix/sleeper-thread-watchdog branch from fd39262 to 6c4b8f0 Compare November 2, 2026 07:26
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant