perf: speed up runtime pruning and run-end execution

[joseph-isaacs]

Summary

Apply supported join runtime filters before Vortex-to-Arrow conversion, and avoid decoding the same compressed run-end page at every binary-search comparison. Default Vortex retains the runtime-pruning gains while Compact recovers the regressions caused by repeated PCO decoding.

Changes

• Cache versioned runtime predicates and shared file split boundaries in the persistent DataFusion scan; combine masks before gathering projected values.
• Normalize integer membership sets once and speed up nullable primitive and boolean run-end decoding. Keep the nullable helper out of the non-nullable decode loop.
• Opt in to retained compressed probes for run-end boundary searches and read primitive ends directly. Preserve Send for the existing sorted-search adapter, with a compile-time regression test.
• Cover PCO searches across pages, duplicate values, null prefixes, and sliced inputs; refresh SQL explain snapshots for runtime-filter placement.

Results

All 99 TPC-DS SF1 queries completed locally and on CI for the current 3850f40 source versus exact base 731a2314. Negative values mean lower geometric mean runtime. The CI report labels the overall result no clear signal, low confidence; these are raw hot-median comparisons.

Engine	Format	CI runtime vs base	Local runtime vs base
DataFusion	default Vortex	-3.9%	-8.3%
DataFusion	Compact	-2.7%	-8.3%
DuckDB	default Vortex	-1.0%	-0.4%
DuckDB	Compact	-17.4%	-25.0%

Parquet controls are within 0.3% on CI and 0.4% locally. Local measurements use 30 workers, 94 DataFusion partitions, ARM64, and exactly the same input files for both revisions; hot medians discard the first of eleven iterations. CI uses 94 CPUs, native CPU compilation, a cached baseline, and hot medians discarding the first of five iterations.

DataFusion Compact Q92 remains a regression: about 15% in the final local run, 13% in a separate 21-iteration repeat, and 25% in the current CI run. The full tables retain every query, including regressions. Paired Q92 profiles show additional PCO decoding and integer-membership work. Matching 250-iteration Q19 profiles isolate the retained-probe change: sampled CPU cycles fall about 59%, with approximately 81% less work in the largest PCO u32 decoding stage.

Validation

• Main CI passes on 3850f40, including SQL logic tests, Rust linting, and integrations.
• 68 targeted tests pass: primitive search conformance, PCO repeated searches, the adapter's Send contract, and run-end slicing.
• Clippy passes for vortex-array, vortex-pco, and vortex-runend, all targets and features, with -D warnings; formatting uses nightly-2026-09-10.
• All 13 suites in the full SQL matrix and the targeted FineWeb S3 repeat completed successfully on 3850f40.
• Codspeed's benchmark workflow succeeds, but its separate Performance Analysis check reports an internal provider processing error. This does not establish performance acceptance.

All 99 CI query runtimes — default Vortex and Compact

TPC-DS SF1 — default Vortex and Compact, all 99 queries

PR 3850f40afa333abd3127f1358a4811131eddc3e3 versus 731a23142e9dc73345f7f79ecc8b665f1d49ba32. Official CI report.

All runtimes are hot medians in milliseconds. Negative changes mean faster. Official verdict: No clear signal (low confidence).

Datafusion

Query	Default base	Default PR	Δ	Compact base	Compact PR	Δ
1	24.48	24.03	-1.8%	26.00	26.18	+0.7%
2	46.15	42.92	-7.0%	48.67	43.57	-10.5%
3	11.54	11.46	-0.7%	12.96	13.15	+1.5%
4	113.24	103.41	-8.7%	123.51	112.75	-8.7%
5	59.46	58.46	-1.7%	62.01	60.13	-3.0%
6	23.65	21.88	-7.5%	25.90	24.79	-4.3%
7	33.01	27.67	-16.2%	35.03	29.63	-15.4%
8	25.36	25.72	+1.4%	30.76	33.38	+8.5%
9	16.61	17.04	+2.6%	21.84	22.60	+3.5%
10	28.78	29.78	+3.5%	37.65	36.34	-3.5%
11	62.03	52.95	-14.6%	69.10	60.05	-13.1%
12	16.23	15.45	-4.8%	20.68	21.69	+4.9%
13	33.26	33.79	+1.6%	55.28	47.84	-13.5%
14	90.43	91.01	+0.6%	97.82	96.47	-1.4%
15	23.99	22.23	-7.4%	27.81	24.61	-11.5%
16	20.08	21.58	+7.4%	23.40	23.04	-1.6%
17	62.32	54.62	-12.4%	62.38	66.01	+5.8%
18	69.14	50.37	-27.2%	69.34	54.81	-21.0%
19	18.14	17.78	-2.0%	22.37	24.19	+8.1%
20	17.65	17.07	-3.3%	22.11	21.92	-0.9%
21	17.65	16.49	-6.5%	18.27	16.63	-9.0%
22	49.46	49.38	-0.2%	50.30	49.40	-1.8%
23	72.50	72.49	-0.0%	77.81	76.73	-1.4%
24	70.27	65.29	-7.1%	77.75	77.74	-0.0%
25	48.09	47.74	-0.7%	55.83	53.13	-4.8%
26	30.32	29.83	-1.6%	33.26	30.78	-7.4%
27	62.98	55.85	-11.3%	63.23	58.91	-6.8%
28	18.66	17.79	-4.7%	26.69	27.11	+1.6%
29	60.29	53.64	-11.0%	68.32	62.95	-7.9%
30	27.56	27.91	+1.3%	30.20	31.68	+4.9%
31	64.47	65.22	+1.2%	76.02	85.11	+12.0%
32	13.50	12.67	-6.1%	16.63	15.28	-8.1%
33	26.95	26.75	-0.7%	28.96	27.30	-5.7%
34	18.34	18.08	-1.5%	21.56	20.36	-5.6%
35	34.59	36.45	+5.4%	42.44	41.25	-2.8%
36	48.74	46.92	-3.7%	51.20	52.21	+2.0%
37	13.18	13.33	+1.1%	15.77	16.65	+5.6%
38	29.28	29.99	+2.4%	33.48	33.28	-0.6%
39	65.58	62.55	-4.6%	66.98	63.61	-5.0%
40	24.87	22.70	-8.7%	28.06	25.20	-10.2%
41	16.26	16.56	+1.8%	18.51	18.81	+1.6%
42	11.40	11.20	-1.8%	12.40	13.15	+6.1%
43	13.18	13.40	+1.6%	15.33	16.60	+8.3%
44	26.78	26.42	-1.4%	30.28	26.82	-11.4%
45	27.71	25.59	-7.7%	31.44	30.06	-4.4%
46	24.83	23.68	-4.7%	28.56	30.15	+5.6%
47	76.85	75.20	-2.1%	89.15	86.99	-2.4%
48	31.76	30.77	-3.1%	51.90	44.08	-15.1%
49	60.70	60.38	-0.5%	67.78	62.64	-7.6%
50	34.57	31.84	-7.9%	36.13	33.08	-8.4%
51	60.30	60.29	-0.0%	63.40	63.81	+0.7%
52	12.09	11.97	-1.0%	13.23	12.87	-2.7%
53	18.47	18.10	-2.0%	23.16	23.40	+1.0%
54	34.07	32.89	-3.5%	40.97	35.38	-13.6%
55	11.48	11.39	-0.7%	13.10	12.75	-2.6%
56	26.97	26.57	-1.5%	28.28	28.49	+0.7%
57	72.81	74.63	+2.5%	79.57	78.55	-1.3%
58	40.51	37.55	-7.3%	47.55	47.56	+0.0%
59	32.70	27.27	-16.6%	41.52	33.97	-18.2%
60	27.13	26.68	-1.7%	28.75	29.25	+1.7%
61	24.74	22.23	-10.1%	27.26	28.32	+3.9%
62	17.34	15.83	-8.7%	17.90	19.72	+10.2%
63	18.48	18.21	-1.4%	21.79	21.56	-1.0%
64	322.92	286.24	-11.4%	447.95	406.87	-9.2%
65	76.35	76.17	-0.2%	81.91	82.69	+0.9%
66	52.38	51.70	-1.3%	54.73	51.84	-5.3%
67	55.53	55.78	+0.5%	59.62	59.44	-0.3%
68	23.70	23.27	-1.8%	30.13	31.50	+4.5%
69	27.61	28.18	+2.1%	36.24	36.21	-0.1%
70	71.82	71.03	-1.1%	76.52	76.22	-0.4%
71	19.69	18.79	-4.6%	20.65	21.23	+2.8%
72	465.72	282.23	-39.4%	463.60	280.05	-39.6%
73	17.38	16.69	-3.9%	19.78	19.88	+0.5%
74	45.60	41.72	-8.5%	51.64	48.55	-6.0%
75	129.30	130.05	+0.6%	136.56	134.62	-1.4%
76	43.78	44.11	+0.7%	41.44	41.92	+1.2%
77	34.68	35.84	+3.3%	37.48	39.10	+4.3%
78	80.71	70.81	-12.3%	87.05	75.50	-13.3%
79	21.06	21.27	+1.0%	24.97	23.69	-5.1%
80	76.36	70.56	-7.6%	78.87	74.08	-6.1%
81	27.68	28.85	+4.2%	29.63	31.44	+6.1%
82	13.72	13.23	-3.5%	15.74	16.67	+6.0%
83	36.25	34.46	-4.9%	37.34	37.65	+0.8%
84	11.54	12.39	+7.4%	12.96	14.10	+8.8%
85	59.46	55.53	-6.6%	103.48	100.40	-3.0%
86	16.29	16.55	+1.6%	16.84	18.49	+9.8%
87	29.71	29.34	-1.3%	33.51	31.68	-5.4%
88	34.61	30.65	-11.5%	40.33	40.39	+0.1%
89	20.80	20.95	+0.7%	24.73	25.70	+3.9%
90	10.38	10.07	-2.9%	11.31	10.79	-4.6%
91	16.40	16.39	-0.1%	27.46	24.32	-11.4%
92	13.47	13.76	+2.2%	14.79	18.54	+25.3%
93	24.23	23.93	-1.3%	25.05	24.69	-1.4%
94	19.66	19.74	+0.4%	21.81	22.41	+2.8%
95	45.98	42.23	-8.2%	49.58	46.04	-7.1%
96	9.59	8.52	-11.1%	11.66	10.93	-6.3%
97	21.52	20.92	-2.7%	24.91	22.51	-9.6%
98	16.93	15.60	-7.9%	22.01	21.48	-2.4%
99	19.40	19.71	+1.6%	21.59	21.64	+0.2%

Duckdb

Query	Default base	Default PR	Δ	Compact base	Compact PR	Δ
1	15.40	15.42	+0.1%	16.62	16.68	+0.4%
2	14.87	15.05	+1.2%	16.37	15.26	-6.8%
3	12.63	10.13	-19.8%	21.12	12.16	-42.4%
4	50.31	50.05	-0.5%	54.03	52.44	-2.9%
5	18.91	18.62	-1.6%	24.15	21.28	-11.9%
6	28.69	26.53	-7.5%	33.69	28.56	-15.2%
7	15.58	15.81	+1.5%	25.17	20.59	-18.2%
8	18.91	18.99	+0.4%	49.38	26.30	-46.7%
9	13.71	13.58	-0.9%	19.06	19.27	+1.1%
10	27.37	27.22	-0.6%	45.38	33.73	-25.7%
11	40.08	40.73	+1.6%	45.78	42.69	-6.8%
12	10.25	10.26	+0.1%	13.37	13.47	+0.8%
13	21.41	21.27	-0.6%	36.37	25.40	-30.2%
14	58.64	58.32	-0.5%	66.71	61.54	-7.7%
15	20.86	20.69	-0.8%	26.10	23.27	-10.9%
16	17.16	17.08	-0.4%	24.84	19.62	-21.0%
17	30.16	29.72	-1.5%	44.60	34.46	-22.7%
18	28.86	27.83	-3.6%	35.61	31.53	-11.5%
19	23.47	23.63	+0.7%	35.43	27.53	-22.3%
20	10.79	10.59	-1.9%	14.02	13.08	-6.7%
21	10.98	10.79	-1.7%	10.67	10.52	-1.4%
22	29.26	29.13	-0.4%	30.95	30.52	-1.4%
23	37.68	37.58	-0.3%	44.14	39.91	-9.6%
24	27.76	27.96	+0.7%	39.54	31.07	-21.4%
25	25.15	25.70	+2.2%	43.23	30.04	-30.5%
26	13.39	13.62	+1.7%	22.41	17.79	-20.6%
27	18.24	18.45	+1.1%	29.34	21.56	-26.5%
28	12.35	11.65	-5.6%	23.44	23.27	-0.7%
29	28.71	28.39	-1.1%	44.29	33.05	-25.4%
30	19.58	19.37	-1.1%	22.52	22.64	+0.6%
31	19.19	19.11	-0.4%	23.69	20.77	-12.3%
32	9.95	9.84	-1.1%	16.40	11.83	-27.8%
33	17.04	16.69	-2.1%	23.05	18.73	-18.8%
34	15.76	15.53	-1.4%	21.87	16.98	-22.4%
35	46.35	45.67	-1.5%	61.91	51.48	-16.9%
36	15.69	14.87	-5.2%	22.64	18.84	-16.8%
37	13.95	13.59	-2.6%	18.99	17.24	-9.2%
38	22.76	21.99	-3.4%	26.43	23.77	-10.0%
39	18.53	18.29	-1.3%	22.33	18.58	-16.8%
40	13.18	12.98	-1.5%	16.09	14.56	-9.5%
41	6.24	6.21	-0.6%	7.42	7.30	-1.6%
42	8.31	8.09	-2.6%	11.83	10.33	-12.7%
43	10.16	10.12	-0.4%	17.34	13.37	-22.9%
44	15.47	15.76	+1.9%	22.94	18.64	-18.8%
45	22.25	22.18	-0.3%	27.41	25.65	-6.4%
46	18.53	18.68	+0.8%	29.21	20.20	-30.9%
47	31.91	30.28	-5.1%	39.89	32.14	-19.4%
48	19.24	19.41	+0.9%	33.65	23.96	-28.8%
49	16.46	16.70	+1.5%	24.00	21.37	-10.9%
50	16.53	16.44	-0.5%	27.75	18.72	-32.5%
51	64.26	65.11	+1.3%	54.58	53.06	-2.8%
52	8.49	8.43	-0.8%	12.90	9.77	-24.3%
53	14.19	14.47	+2.0%	21.46	16.32	-23.9%
54	20.31	20.69	+1.9%	27.58	22.04	-20.1%
55	8.43	8.33	-1.1%	12.01	9.77	-18.6%
56	17.02	17.19	+1.0%	24.63	20.25	-17.8%
57	24.01	24.11	+0.4%	28.16	25.27	-10.3%
58	17.75	17.41	-1.9%	23.93	19.30	-19.4%
59	18.79	18.71	-0.4%	23.29	19.13	-17.9%
60	18.05	18.21	+0.9%	24.75	19.50	-21.2%
61	19.40	19.48	+0.4%	31.34	23.34	-25.5%
62	10.15	10.07	-0.8%	13.34	11.55	-13.4%
63	13.23	12.94	-2.2%	21.49	15.11	-29.7%
64	62.93	63.40	+0.8%	85.92	70.57	-17.9%
65	15.64	14.81	-5.3%	19.76	18.24	-7.7%
66	21.11	21.33	+1.1%	25.22	22.76	-9.7%
67	66.35	65.34	-1.5%	72.73	66.94	-8.0%
68	18.31	18.33	+0.1%	28.87	20.05	-30.5%
69	28.46	28.67	+0.7%	46.49	34.84	-25.1%
70	19.54	18.32	-6.3%	31.30	20.99	-32.9%
71	12.57	12.52	-0.4%	19.70	14.87	-24.5%
72	62.21	61.54	-1.1%	69.88	64.91	-7.1%
73	15.19	14.51	-4.4%	22.07	16.19	-26.6%
74	27.39	26.86	-2.0%	32.09	28.91	-9.9%
75	29.16	28.80	-1.2%	33.12	31.41	-5.2%
76	12.73	12.62	-0.9%	16.23	13.96	-14.0%
77	16.10	16.25	+1.0%	22.48	18.17	-19.1%
78	37.20	36.82	-1.0%	42.31	38.98	-7.9%
79	14.64	14.36	-1.9%	37.51	16.75	-55.4%
80	27.35	28.44	+4.0%	37.92	30.77	-18.9%
81	22.40	22.69	+1.3%	24.20	24.43	+1.0%
82	32.22	32.32	+0.3%	35.86	35.09	-2.2%
83	19.72	19.19	-2.7%	29.98	20.75	-30.8%
84	12.85	12.63	-1.7%	14.75	14.71	-0.3%
85	30.81	30.61	-0.7%	41.46	36.57	-11.8%
86	10.81	10.66	-1.4%	12.15	12.20	+0.4%
87	24.02	24.13	+0.4%	28.40	25.27	-11.0%
88	32.43	31.92	-1.6%	66.72	41.47	-37.8%
89	13.74	13.85	+0.8%	21.33	15.46	-27.5%
90	8.15	8.12	-0.5%	12.41	9.59	-22.8%
91	15.29	15.11	-1.2%	30.59	25.02	-18.2%
92	12.79	11.96	-6.5%	20.17	15.06	-25.3%
93	17.41	16.80	-3.5%	18.28	17.76	-2.8%
94	16.28	16.02	-1.6%	21.94	18.69	-14.8%
95	44.45	45.48	+2.3%	46.94	44.67	-4.8%
96	7.17	7.34	+2.5%	16.91	8.99	-46.8%
97	22.46	22.13	-1.5%	25.17	22.89	-9.0%
98	13.32	13.16	-1.2%	18.01	16.23	-9.9%
99	13.24	13.59	+2.6%	16.89	16.20	-4.1%

All 13 SQL suites — original full run, hot runtime comparisons

Full SQL CI results for 3850f40afa333abd3127f1358a4811131eddc3e3 versus 731a23142e9dc73345f7f79ecc8b665f1d49ba32.

Raw hot median geometric mean runtime changes; negative means faster. The official reports separately estimate confidence and environmental drift.

Suite	DF default	DF Compact	DuckDB default	DuckDB Compact	Official verdict
PolarSignals Profiling	+0.4%	—	—	—	No attributed verdict
FineWeb NVMe	-1.5%	+0.1%	-1.4%	-5.1%	No clear signal (low confidence)
TPC-H SF=1 on NVME	+1.4%	+1.0%	+1.0%	-1.4%	No clear signal (low confidence)
FineWeb S3 (original)	+35.0%	-3.4%	-3.4%	-3.8%	No clear signal (environment too noisy confidence)
Clickbench Sorted on NVME	-1.3%	—	-4.6%	—	No clear signal (low confidence)
Appian on NVME	-1.3%	—	-0.4%	—	No clear signal (low confidence)
Clickbench on NVME	+0.1%	—	-0.7%	—	No clear signal (low confidence)
TPC-H SF=10 on NVME	-1.1%	-1.1%	+0.8%	-1.8%	No clear signal (low confidence)
TPC-DS SF=1 on NVME	-3.9%	-2.7%	-1.0%	-17.4%	No clear signal (low confidence)
Statistical and Population Genetics	+1.3%	-8.1%	-0.4%	-5.5%	No clear signal (low confidence)
TPC-H SF=1 on S3	-4.6%	-2.0%	-5.5%	-2.7%	No clear signal (environment too noisy confidence)
Vortex queries	+0.3%	—	+0.6%	—	No clear signal (low confidence)
TPC-H SF=10 on S3	-8.0%	+0.2%	-6.7%	-7.8%	No clear signal (environment too noisy confidence)

The FineWeb S3 DataFusion default outlier did not repeat. Both runs use the same source and cached base; both reports label the environment too noisy for confident attribution. The original full-run measurements remain above. The report comment now displays the targeted repeat.

Engine	Format	Original FineWeb S3 runtime vs base	Repeat runtime vs base
DataFusion	default Vortex	+35.0%	-7.9%
DataFusion	Compact	-3.4%	-15.4%
DuckDB	default Vortex	-3.4%	-4.7%
DuckDB	Compact	-3.8%	-9.8%

These measurements show variability; they do not establish a confident S3 performance improvement.