hypatia-bench exists so that a green run predicts a green pull request. It keeps
its own copy of the list it is being green about.
ci.yaml's rust-tests job runs five recipes:
just rust-test-core
just rust-test-db
just rust-test-ingest
just rust-test-api-unit
just rust-test-api-integration
bin/hypatia-bench runs five commands of its own that happen to mean the same
thing today. The equivalence is hand-maintained, and it has already broken once:
the bench named cargo test -p api-rest --test integration until b3f2ec7, long
after that binary stopped existing — the integration suite had been split into
one file per surface, and a gate naming a target that no longer exists is a gate
that runs nothing. Nothing reports the next divergence either, in either
direction: a recipe that changes what it runs, or a step added to the job, is
invisible here because the two lists never meet.
Fix: read the commands out of ci.yaml and run those, the way
bin/hypatia-compose-check already reads the compose invocation, instead of
keeping a second spelling. Refuse by name when the extraction comes back empty,
so an unparsed workflow cannot pass for an empty gate list.
The scope is the cargo test block. The steps around it — cargo check, the
migrate run, the table-count assertion, the test-case registration — differ from
ci.yaml deliberately, for the reasons their comments give, and reading those out
of the workflow would be wrong rather than merely different.
just is installed on the build host (mise, 1.36.0) but is absent from the PATH
a non-interactive ssh gets: ~/.local/share/mise/shims/just, or mise which just.
Done when: the bench runs the rust test list ci.yaml names, names each
command it runs, and fails when a test file under crates/api-rest/tests/ fails.
hypatia-benchexists so that a green run predicts a green pull request. It keepsits own copy of the list it is being green about.
ci.yaml's
rust-testsjob runs five recipes:bin/hypatia-benchruns five commands of its own that happen to mean the samething today. The equivalence is hand-maintained, and it has already broken once:
the bench named
cargo test -p api-rest --test integrationuntil b3f2ec7, longafter that binary stopped existing — the integration suite had been split into
one file per surface, and a gate naming a target that no longer exists is a gate
that runs nothing. Nothing reports the next divergence either, in either
direction: a recipe that changes what it runs, or a step added to the job, is
invisible here because the two lists never meet.
Fix: read the commands out of ci.yaml and run those, the way
bin/hypatia-compose-checkalready reads the compose invocation, instead ofkeeping a second spelling. Refuse by name when the extraction comes back empty,
so an unparsed workflow cannot pass for an empty gate list.
The scope is the
cargo testblock. The steps around it —cargo check, themigrate run, the table-count assertion, the test-case registration — differ from
ci.yaml deliberately, for the reasons their comments give, and reading those out
of the workflow would be wrong rather than merely different.
justis installed on the build host (mise, 1.36.0) but is absent from the PATHa non-interactive ssh gets:
~/.local/share/mise/shims/just, ormise which just.Done when: the bench runs the rust test list ci.yaml names, names each
command it runs, and fails when a test file under
crates/api-rest/tests/fails.