Skip to content

bench(array): measure RowFn Boolean retry in CI - #9988

Merged
myrrc merged 5 commits into
developfrom
ct/row-fn-bool-bench
Sep 23, 2026
Merged

myrrc merged 5 commits into
developfrom
ct/row-fn-bool-bench

Conversation

@connortsui20

@connortsui20 connortsui20 commented Sep 22, 2026 •

Copy link
Copy Markdown
Member

Summary

Adds a deferred Boolean RowFn benchmark to establish the CI baseline for #9986. It uses the multiversioned collector over partially valid input and covers four cases: an accepted dense attempt and a null-only retry, each with two columns and with a constant left operand against a column.

Changes

Uses the existing native AVX2, AVX-512, and NEON jobs, so there are 12 benchmarks in total. Each timed iteration executes eight 16,384-row batches to clear the wall-time floor. Fixture and execution-context setup and result destruction stay outside timing. Backtraces follow the CI setting. This layer changes no production code.

An earlier revision ran 72 cases (both collector flags, constants in either position, all-valid input, and observable failures). All of them completed in native CI, with medians from 32.31 to 490.6 µs and a maximum printed sample of 712.1 µs.

Comparison with #9986 and raw results come from that earlier revision.

@codspeed

codspeed Bot commented Sep 22, 2026 •

Copy link
Copy Markdown

Merging this PR will regress 1 benchmark

⚠️ Unknown Walltime execution environment detected

Using the Walltime instrument on standard Hosted Runners will lead to inconsistent data.

For the most accurate results, we recommend using CodSpeed Macro Runners: bare-metal machines fine-tuned for performance measurement consistency.

⚠️ 4 benchmarks measured no execution time

Nothing ran under measurement, usually because the compiler removed the code under test. These results are not comparable, so they count as unchanged.

Preventing compiler optimizations

⚠️ Different runtime environments detected

Some benchmarks with significant performance changes were compared across different runtime environments,
which may affect the accuracy of the results.

Open the report in CodSpeed to investigate

⚡ 4 improved benchmarks
❌ 1 regressed benchmark
✅ 2203 untouched benchmarks
🆕 24 new benchmarks
⏩ 329 skipped benchmarks1

Warning

Please fix the performance issues or acknowledge them on CodSpeed.

Performance Changes

Mode Benchmark BASE HEAD Efficiency
❌ WallTime scalar_subtract_neon 12.3 µs 13.7 µs -10.14%
⚡ Simulation take_fsl_nullable_random[16, 100] 97.1 µs 50.4 µs +92.78%
⚡ Simulation take_fsl_f16_random[256, 100] 141.7 µs 85.7 µs +65.33%
⚡ Simulation take_chunked_fsl_sorted[32, 64] 252.9 µs 196.6 µs +28.62%
⚡ Simulation take_fsl_random[64, 100] 144.5 µs 125.4 µs +15.2%
🆕 WallTime decode_avx2[8192, (External, AllValid)] N/A 166.2 µs N/A
🆕 WallTime decode_avx2[8192, (External, OneNullInEight)] N/A 147.1 µs N/A
🆕 WallTime decode_avx2[8192, (Inline, AllValid)] N/A 102.9 µs N/A
🆕 WallTime decode_avx2[8192, (Inline, OneNullInEight)] N/A 95.1 µs N/A
🆕 WallTime deferred_bool_avx2[16384, (Columns, NullOnlyFailure)] N/A 344.8 µs N/A
🆕 WallTime deferred_bool_avx2[16384, (Columns, PartialAccepted)] N/A 110.2 µs N/A
🆕 WallTime deferred_bool_avx2[16384, (ConstantLhs, NullOnlyFailure)] N/A 454.6 µs N/A
🆕 WallTime deferred_bool_avx2[16384, (ConstantLhs, PartialAccepted)] N/A 206.3 µs N/A
🆕 WallTime decode_avx512[8192, (External, AllValid)] N/A 171.1 µs N/A
🆕 WallTime decode_avx512[8192, (External, OneNullInEight)] N/A 153.3 µs N/A
🆕 WallTime decode_avx512[8192, (Inline, AllValid)] N/A 99.7 µs N/A
🆕 WallTime decode_avx512[8192, (Inline, OneNullInEight)] N/A 89.8 µs N/A
🆕 WallTime deferred_bool_avx512[16384, (Columns, NullOnlyFailure)] N/A 275.3 µs N/A
🆕 WallTime deferred_bool_avx512[16384, (Columns, PartialAccepted)] N/A 42.3 µs N/A
🆕 WallTime deferred_bool_avx512[16384, (ConstantLhs, NullOnlyFailure)] N/A 469.5 µs N/A
... ... ... ... ... ...

ℹ️ Only the first 20 benchmarks are displayed. Go to the app to view all benchmarks.

Tip

Investigate this regression by commenting @codspeedbot fix this regression on this PR, or directly use the CodSpeed MCP with your agent.


Comparing ct/row-fn-bool-bench (81ace30) with develop (36cb75d)2

Open in CodSpeed

Footnotes

  1. 329 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports. ↩

  2. No successful run was found on develop (1eb5b43) during the generation of this report, so 36cb75d was used instead as the comparison base. There might be some changes unrelated to this pull request in this report. ↩

@connortsui20 connortsui20 added the changelog/skip Do not list PR in the changelog label Sep 22, 2026
@connortsui20
connortsui20 force-pushed the ct/row-fn-bool-bench branch 2 times, most recently from 1d65020 to f812b78 Compare September 23, 2026 02:01
@connortsui20
connortsui20 marked this pull request as ready for review September 23, 2026 02:01
connortsui20 and others added 5 commits September 23, 2026 02:32
Signed-off-by: Connor Tsui <connor.tsui20@gmail.com>
Signed-off-by: Connor Tsui <connor.tsui20@gmail.com>
Keep only two-column inputs and a constant left operand against a column.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015jJxqdFc4B1PsoctjdFZhd
Signed-off-by: Claude <noreply@anthropic.com>
Keep only the multiversioned collector over partially valid input, for
an accepted dense attempt and a null-only retry, each with two columns
and with a constant left operand.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015jJxqdFc4B1PsoctjdFZhd
Signed-off-by: Claude <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015jJxqdFc4B1PsoctjdFZhd
Signed-off-by: Claude <noreply@anthropic.com>
@myrrc
myrrc merged commit 9ac9454 into develop Sep 23, 2026
86 of 87 checks passed
@myrrc
myrrc deleted the ct/row-fn-bool-bench branch September 23, 2026 09:27
connortsui20 added a commit that referenced this pull request Sep 23, 2026
Stacked on #9988, which adds the CI benchmark without changing
production code.

## Summary

Routes deferred Boolean dense attempts through the packed collector,
preserving the requested multiversioning flag. Keeps the source and
failure accumulator in one borrowed state, and inlines the collector's
scalar tail so LLVM can retain the alias information needed to vectorize
the row loop. Decode errors remain terminal, and null-only failures
still retry valid rows.

The earlier x86 regressions came from scalarized collector loops.
Optimized Linux IR and the native CI executables confirm that these
loops now vectorize.

## Changes

Native CI medians in microseconds per eight 16,384-row batches, with
partial validity and the multiversioned collector:

| Target | Columns before → after | Constant LHS before → after |
| --- | ---: | ---: |
| AVX2 | 111.0 → 44.53 | 217.1 → 34.58 |
| AVX-512 | 42.81 → 45.07 | 206.9 → 37.25 |
| NEON | 68.32 → 71.01 | 173.3 → 57.62 |

Constant RHS also improves on every target. Small column-only movements
remain uncertain after one paired run. All final cases are below 1 ms,
with a maximum printed sample of 555.8 µs.

Passed 175 RowFn tests, 14 Boolean packing tests, and scoped Clippy. The
CodSpeed check flags no Boolean retry regressions, but remains red on
three other, unattributed benchmark alerts. This remains a draft.

[Four-stage comparison, compiler evidence, and raw
results](https://github.com/vortex-data/vortex/blob/ct/row-fn-performance-evidence/research/row-fn-engine/performance/ci-bool-capture.md).
Evidence stays on the separate branch.

Signed-off-by: Connor Tsui <connor.tsui20@gmail.com>
haydenflinner pushed a commit to haydenflinner/vortex that referenced this pull request Oct 14, 2026
…a#9986)

Stacked on vortex-data#9988, which adds the CI benchmark without changing
production code.

## Summary

Routes deferred Boolean dense attempts through the packed collector,
preserving the requested multiversioning flag. Keeps the source and
failure accumulator in one borrowed state, and inlines the collector's
scalar tail so LLVM can retain the alias information needed to vectorize
the row loop. Decode errors remain terminal, and null-only failures
still retry valid rows.

The earlier x86 regressions came from scalarized collector loops.
Optimized Linux IR and the native CI executables confirm that these
loops now vectorize.

## Changes

Native CI medians in microseconds per eight 16,384-row batches, with
partial validity and the multiversioned collector:

| Target | Columns before → after | Constant LHS before → after |
| --- | ---: | ---: |
| AVX2 | 111.0 → 44.53 | 217.1 → 34.58 |
| AVX-512 | 42.81 → 45.07 | 206.9 → 37.25 |
| NEON | 68.32 → 71.01 | 173.3 → 57.62 |

Constant RHS also improves on every target. Small column-only movements
remain uncertain after one paired run. All final cases are below 1 ms,
with a maximum printed sample of 555.8 µs.

Passed 175 RowFn tests, 14 Boolean packing tests, and scoped Clippy. The
CodSpeed check flags no Boolean retry regressions, but remains red on
three other, unattributed benchmark alerts. This remains a draft.

[Four-stage comparison, compiler evidence, and raw
results](https://github.com/vortex-data/vortex/blob/ct/row-fn-performance-evidence/research/row-fn-engine/performance/ci-bool-capture.md).
Evidence stays on the separate branch.

Signed-off-by: Connor Tsui <connor.tsui20@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

changelog/skip Do not list PR in the changelog

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants