From 8caf019faed6cd46443c42e58e4c3e24d7bf73d6 Mon Sep 17 00:00:00 2001 From: Hongquan Li Date: Sun, 12 Jul 2026 16:25:22 -0700 Subject: [PATCH 1/6] fix(ci): run GUI test under pyqt5 instead of skipping test_HighContentScreeningGui.py has been --ignore'd in CI since #391 due to a PySide6 QObject double-init failure. The known-good local recipe (force qtpy/pytest-qt onto PyQt5) works headless, and setup_22.04.sh already installs PyQt5 via apt (python3-pyqt5), so pin QT_API/PYTEST_QT_API=pyqt5 at the job level and drop the --ignore. Verified locally headless under the pin: 5 passed in 17.9s. The PySide6 double-init itself remains unfixed (tracked separately). Co-Authored-By: Claude Fable 5 --- .github/workflows/main.yml | 8 ++++++-- 1 file changed, 6 insertions(+), 2 deletions(-) diff --git a/.github/workflows/main.yml b/.github/workflows/main.yml index 2a4fa735f..bdd94c9bd 100644 --- a/.github/workflows/main.yml +++ b/.github/workflows/main.yml @@ -11,6 +11,11 @@ jobs: runs-on: ubuntu-22.04 env: DISPLAY: ":99.0" + # Force qtpy/pytest-qt onto PyQt5 (installed via apt in setup_22.04.sh). + # Under PySide6 the GUI test fails with a QObject double-init error; + # root cause unresolved — see the PR that added this pin. + QT_API: pyqt5 + PYTEST_QT_API: pyqt5 steps: # Our workflows were running out of memory. We don't need most of the pre-installed machinery in the # github ubuntu image, so remove it using this helper action. @@ -52,6 +57,5 @@ jobs: sudo apt install libxkbcommon-x11-0 xvfb libxcb-icccm4 libxcb-image0 libxcb-keysyms1 libxcb-randr0 libxcb-render-util0 libxcb-xinerama0 libxcb-xfixes0 x11-utils /sbin/start-stop-daemon --start --quiet --pidfile /tmp/custom_xvfb_99.pid --make-pidfile --background --exec /usr/bin/Xvfb -- :99 -screen 0 1920x1200x24 -ac +extension GLX - name: Run the tests - # test_HighContentScreeningGui.py: PySide6 QObject double-init bug (separate issue) - run: python3 -m pytest --ignore=tests/control/test_HighContentScreeningGui.py + run: python3 -m pytest working-directory: ./software From fd8bf5a856dd4f458b61181418919b6163c2f148 Mon Sep 17 00:00:00 2001 From: Hongquan Li Date: Sun, 12 Jul 2026 16:49:05 -0700 Subject: [PATCH 2/6] fix(ci): isolate the GUI test in its own pytest process MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit First CI run showed the pyqt5 pin works (the GUI test passed), but running it in the shared pytest process deterministically segfaults a later Qt test (test_channel_sequence.py:116 qtbot.waitUntil) — reproduced locally on macOS with the same two-file ordering (exit 139). The GUI test leaves Qt/thread state behind (faulthandler dump shows the simulated-camera stream thread and slack-notifier worker still alive). Keep the main run ignoring it and add a second pytest invocation for the GUI test alone. Co-Authored-By: Claude Fable 5 --- .github/workflows/main.yml | 8 +++++++- 1 file changed, 7 insertions(+), 1 deletion(-) diff --git a/.github/workflows/main.yml b/.github/workflows/main.yml index bdd94c9bd..890d434c5 100644 --- a/.github/workflows/main.yml +++ b/.github/workflows/main.yml @@ -57,5 +57,11 @@ jobs: sudo apt install libxkbcommon-x11-0 xvfb libxcb-icccm4 libxcb-image0 libxcb-keysyms1 libxcb-randr0 libxcb-render-util0 libxcb-xinerama0 libxcb-xfixes0 x11-utils /sbin/start-stop-daemon --start --quiet --pidfile /tmp/custom_xvfb_99.pid --make-pidfile --background --exec /usr/bin/Xvfb -- :99 -screen 0 1920x1200x24 -ac +extension GLX - name: Run the tests - run: python3 -m pytest + run: python3 -m pytest --ignore=tests/control/test_HighContentScreeningGui.py + working-directory: ./software + - name: Run the GUI test in its own process + # The full-application GUI test leaves Qt/thread state behind that + # deterministically segfaults later Qt tests (e.g. test_channel_sequence) + # when run in the same pytest process, so it gets its own invocation. + run: python3 -m pytest tests/control/test_HighContentScreeningGui.py working-directory: ./software From 0d3be5692cc2aea8335fee200a1a4b687c5ee5b8 Mon Sep 17 00:00:00 2001 From: Hongquan Li Date: Sun, 12 Jul 2026 17:04:43 -0700 Subject: [PATCH 3/6] fix(ci): scope the pyqt5 pin to the GUI-test step only MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit With the job-level pin, the main pytest run passed all 1413 tests and then segfaulted at interpreter shutdown — the pin flips pytest-qt/qtpy binding selection for the whole suite, hitting PyQt5's destructor-order-vs-GC conflict at exit (the reason main_hcs.py uses os._exit()). Master runs the main suite unpinned and green, so keep it identical and pin only the isolated GUI-test step, which is the invocation that needs it. Co-Authored-By: Claude Fable 5 --- .github/workflows/main.yml | 20 ++++++++++++-------- 1 file changed, 12 insertions(+), 8 deletions(-) diff --git a/.github/workflows/main.yml b/.github/workflows/main.yml index 890d434c5..9b8aa5866 100644 --- a/.github/workflows/main.yml +++ b/.github/workflows/main.yml @@ -11,11 +11,6 @@ jobs: runs-on: ubuntu-22.04 env: DISPLAY: ":99.0" - # Force qtpy/pytest-qt onto PyQt5 (installed via apt in setup_22.04.sh). - # Under PySide6 the GUI test fails with a QObject double-init error; - # root cause unresolved — see the PR that added this pin. - QT_API: pyqt5 - PYTEST_QT_API: pyqt5 steps: # Our workflows were running out of memory. We don't need most of the pre-installed machinery in the # github ubuntu image, so remove it using this helper action. @@ -60,8 +55,17 @@ jobs: run: python3 -m pytest --ignore=tests/control/test_HighContentScreeningGui.py working-directory: ./software - name: Run the GUI test in its own process - # The full-application GUI test leaves Qt/thread state behind that - # deterministically segfaults later Qt tests (e.g. test_channel_sequence) - # when run in the same pytest process, so it gets its own invocation. + # Own invocation: the full-application GUI test leaves Qt/thread state + # behind that deterministically segfaults later Qt tests (e.g. + # test_channel_sequence) when run in the same pytest process. + # Step-scoped pyqt5 pin: without it, pytest-qt picks PySide6 for qtbot + # while the app code (qtpy) uses PyQt5, and the mixed bindings fail with + # a QObject double-init error. The pin stays off the main run because a + # job-wide pin made the otherwise-green suite segfault at interpreter + # shutdown (PyQt5 destructor-order-vs-GC, cf. the os._exit() note in + # main_hcs.py). run: python3 -m pytest tests/control/test_HighContentScreeningGui.py working-directory: ./software + env: + QT_API: pyqt5 + PYTEST_QT_API: pyqt5 From 9de2f061167c32103dc0c149a1e9919612c91d4b Mon Sep 17 00:00:00 2001 From: Hongquan Li Date: Sun, 12 Jul 2026 17:22:06 -0700 Subject: [PATCH 4/6] fix(ci): hard-exit the isolated GUI-test process after a clean session MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Third CI iteration: the isolated GUI-test step passed (5/5) but the process segfaulted during interpreter shutdown — under PyQt5, a process that constructed the full HCS GUI crashes in Qt C++ destructors at exit even when every test passed. This is the same crash main_hcs.py avoids with os._exit(). Add an env-gated (SQUID_PYTEST_HARD_EXIT=1) pytest_unconfigure hook that flushes output and os._exit()s with pytest's real exit status, and set the variable only on the isolated CI step. Verified locally: summary still prints, pass exits 0, failure exits 1. Co-Authored-By: Claude Fable 5 --- .github/workflows/main.yml | 5 +++++ software/tests/control/conftest.py | 21 +++++++++++++++++++++ 2 files changed, 26 insertions(+) diff --git a/.github/workflows/main.yml b/.github/workflows/main.yml index 9b8aa5866..0462df611 100644 --- a/.github/workflows/main.yml +++ b/.github/workflows/main.yml @@ -64,8 +64,13 @@ jobs: # job-wide pin made the otherwise-green suite segfault at interpreter # shutdown (PyQt5 destructor-order-vs-GC, cf. the os._exit() note in # main_hcs.py). + # SQUID_PYTEST_HARD_EXIT: under PyQt5 this process segfaults during + # interpreter shutdown AFTER all tests pass (Qt destructor order vs + # Python GC — the same crash main_hcs.py avoids with os._exit()), so + # the conftest hard-exits with pytest's real status instead. run: python3 -m pytest tests/control/test_HighContentScreeningGui.py working-directory: ./software env: QT_API: pyqt5 PYTEST_QT_API: pyqt5 + SQUID_PYTEST_HARD_EXIT: "1" diff --git a/software/tests/control/conftest.py b/software/tests/control/conftest.py index 485098188..f08f93fef 100644 --- a/software/tests/control/conftest.py +++ b/software/tests/control/conftest.py @@ -6,6 +6,8 @@ """ import logging +import os +import sys from unittest.mock import patch import pytest @@ -16,6 +18,25 @@ logger = logging.getLogger(__name__) +def pytest_sessionfinish(session, exitstatus): + session.config._squid_exitstatus = int(exitstatus) + + +def pytest_unconfigure(config): + """Optionally skip interpreter teardown after the test session. + + Under PyQt5 on Linux, a pytest process that constructed the full HCS GUI + segfaults during interpreter shutdown (Qt C++ destructor order conflicts + with Python GC) even though every test passed. main_hcs.py sidesteps the + same crash with os._exit(); SQUID_PYTEST_HARD_EXIT=1 lets CI's isolated + GUI-test invocation do likewise, preserving pytest's exit status. + """ + if os.environ.get("SQUID_PYTEST_HARD_EXIT") == "1": + sys.stdout.flush() + sys.stderr.flush() + os._exit(getattr(config, "_squid_exitstatus", 0)) + + def _make_tracking_init(original_init, instances_list): """Create a wrapper that tracks Microcontroller instances.""" From de563f898c23418e977d43a84b021163f6d96b73 Mon Sep 17 00:00:00 2001 From: Hongquan Li Date: Sun, 12 Jul 2026 17:52:02 -0700 Subject: [PATCH 5/6] fix(ci): fail loud if the hard-exit hook has no recorded exit status Review finding (Opus): if pytest_sessionfinish never runs (a pytest_sessionstart failure leaves initstate==1, where unconfigure still fires), the previous default of 0 would turn a broken session into a green CI step. Default to 1 so an unrecorded status fails the gate. Co-Authored-By: Claude Fable 5 --- software/tests/control/conftest.py | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/software/tests/control/conftest.py b/software/tests/control/conftest.py index f08f93fef..297f26d27 100644 --- a/software/tests/control/conftest.py +++ b/software/tests/control/conftest.py @@ -34,7 +34,9 @@ def pytest_unconfigure(config): if os.environ.get("SQUID_PYTEST_HARD_EXIT") == "1": sys.stdout.flush() sys.stderr.flush() - os._exit(getattr(config, "_squid_exitstatus", 0)) + # Default 1, not 0: if pytest_sessionfinish never ran (e.g. a + # sessionstart failure), an unrecorded status must fail the step. + os._exit(getattr(config, "_squid_exitstatus", 1)) def _make_tracking_init(original_init, instances_list): From 649c0b8aec48dabacd6384304ae675cd80877a35 Mon Sep 17 00:00:00 2001 From: Hongquan Li Date: Sun, 12 Jul 2026 18:07:30 -0700 Subject: [PATCH 6/6] =?UTF-8?q?fix(ci):=20apply=20the=20hard-exit=20hook?= =?UTF-8?q?=20to=20the=20main=20test=20run=20too=20=E2=80=94=20master's=20?= =?UTF-8?q?suite=20has=20the=20same=20flaky=20exit=20segfault?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The round-4 CI failure on this PR was the main run (byte-identical invocation to master's) segfaulting at interpreter shutdown after 1413 tests passed. Master itself fails the same way — 3 of its last 15 runs: 'N passed' followed by 'Segmentation fault (core dumped)' exit 139 (runs 28883871972, 28762269294, 28559414408). This also retroactively explains this PR's round-2 failure, which was misattributed to the job-level Qt pin. Set SQUID_PYTEST_HARD_EXIT=1 on the main step as well: after the session completes and all reporting is written, the conftest hook os._exit()s with pytest's real status — genuine failures still fail the step, and the ~20%-flaky Qt-teardown crash stops failing green runs. Co-Authored-By: Claude Fable 5 --- .github/workflows/main.yml | 19 ++++++++++--------- 1 file changed, 10 insertions(+), 9 deletions(-) diff --git a/.github/workflows/main.yml b/.github/workflows/main.yml index 0462df611..c0356c628 100644 --- a/.github/workflows/main.yml +++ b/.github/workflows/main.yml @@ -52,22 +52,23 @@ jobs: sudo apt install libxkbcommon-x11-0 xvfb libxcb-icccm4 libxcb-image0 libxcb-keysyms1 libxcb-randr0 libxcb-render-util0 libxcb-xinerama0 libxcb-xfixes0 x11-utils /sbin/start-stop-daemon --start --quiet --pidfile /tmp/custom_xvfb_99.pid --make-pidfile --background --exec /usr/bin/Xvfb -- :99 -screen 0 1920x1200x24 -ac +extension GLX - name: Run the tests + # SQUID_PYTEST_HARD_EXIT: the suite intermittently segfaults during + # interpreter shutdown AFTER all tests pass (Qt destructor order vs + # Python GC — the same crash main_hcs.py avoids with os._exit(); seen + # on master in 3 of the 15 runs before 2026-07-13). The conftest hook + # hard-exits with pytest's real status once the session is complete, + # so real test failures still fail the step. run: python3 -m pytest --ignore=tests/control/test_HighContentScreeningGui.py working-directory: ./software + env: + SQUID_PYTEST_HARD_EXIT: "1" - name: Run the GUI test in its own process # Own invocation: the full-application GUI test leaves Qt/thread state # behind that deterministically segfaults later Qt tests (e.g. # test_channel_sequence) when run in the same pytest process. # Step-scoped pyqt5 pin: without it, pytest-qt picks PySide6 for qtbot - # while the app code (qtpy) uses PyQt5, and the mixed bindings fail with - # a QObject double-init error. The pin stays off the main run because a - # job-wide pin made the otherwise-green suite segfault at interpreter - # shutdown (PyQt5 destructor-order-vs-GC, cf. the os._exit() note in - # main_hcs.py). - # SQUID_PYTEST_HARD_EXIT: under PyQt5 this process segfaults during - # interpreter shutdown AFTER all tests pass (Qt destructor order vs - # Python GC — the same crash main_hcs.py avoids with os._exit()), so - # the conftest hard-exits with pytest's real status instead. + # while the app code (qtpy) uses PyQt5, and the mixed bindings fail + # with a QObject double-init error. run: python3 -m pytest tests/control/test_HighContentScreeningGui.py working-directory: ./software env: