Skip to content

Commit d7f0ec6

Browse files
committed
perf(scan): optimize V2 drivers with V1 cache parity
Avoid repeated V2 plan walks and allocations, group filters sharing columns, preserve pruning masks, and remove decoded segment reuse beyond the V1 policy. Improve portable mask rank scattering and delegate extension BETWEEN predicates to storage so compressed data is decoded once. Cancel skipped DuckDB file preparation and avoid setup on the disabled path. Keep cross-file preparation and bounded projection read-ahead disabled by default. Add opt-in IO/compute timing and paired benchmark, perf, and profile tooling, with the measured results in benchmarks/. The latest non-compact DuckDB comparison reduced Q6 latency by 7.15% and instructions by 21.02%, with unchanged read count and bytes. Q14 improved 6.25% and Q12 improved 2.60%; FineWeb showed no established regression. Release builds, paired perf runs, Samply profiles, IO traces, and the focused BETWEEN microbenchmark completed. Latest regression tests, lint, and formatting were not run locally. Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>
1 parent afeb2d1 commit d7f0ec6

48 files changed

Lines changed: 3996 additions & 286 deletions

File tree

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎.agents/skills/bench-performance/SKILL.md‎

Lines changed: 20 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -72,8 +72,8 @@ Do not wait for a deep code read before showing benchmark comparisons or first s
7272
with the orchestrator or directly:
7373

7474
```bash
75-
cargo build -p datafusion-bench --profile release_debug --features unstable_encodings
76-
cargo build -p duckdb-bench --profile release_debug --features unstable_encodings
75+
cargo build -p datafusion-bench --profile release_debug
76+
cargo build -p duckdb-bench --profile release_debug
7777
cargo build -p lance-bench --profile release_debug
7878
```
7979

@@ -85,6 +85,16 @@ Do not wait for a deep code read before showing benchmark comparisons or first s
8585

8686
## Common Environment
8787

88+
`scripts/bench-v2-drivers.py` compares baseline and candidate binaries for both engines, alternates
89+
run order, excludes warmup, and reports paired round changes. It uses DuckDB's
90+
default fresh connection per iteration and clears segment-cache/preload settings. Its `--perf-stat` runs separate
91+
diagnostic processes with acknowledged perf control FIFOs. The shared runner enables counters only
92+
around query callbacks after `VORTEX_BENCH_PERF_WARMUP` iterations (default 2), using
93+
`VORTEX_BENCH_PERF_CONTROL` and `VORTEX_BENCH_PERF_ACK`. Both binaries must include this support.
94+
DuckDB's callback includes connection reopening unless `--reuse` is specified; perf counts include
95+
that work, while DuckDB's reported query timer excludes it. Use `--reuse` only for an explicitly
96+
labeled profiling run, not as the default performance comparison.
97+
8898
- `RUST_LOG` wins over `--verbose`. Without `RUST_LOG`, `--verbose` raises default logging to
8999
`TRACE`; otherwise the env filter controls output.
90100
- Useful starting filters:
@@ -119,7 +129,10 @@ Supported diagnostics:
119129
- `--explain` prints query plans instead of timing;
120130
- `--show-metrics` prints Vortex execution-plan metrics after a timed run.
121131

122-
Declared but currently not useful unless the source changes: `--threads`, `--emit-plan`, and
132+
`--threads N` sets both Tokio worker threads and DataFusion execution partitions. Without it,
133+
Tokio and DataFusion retain their environment-controlled defaults.
134+
135+
Declared but currently not useful unless the source changes: `--emit-plan` and
123136
`--export-spans` are parsed but not wired into the execution path.
124137

125138
Examples:
@@ -141,6 +154,9 @@ FEATURE_TOGGLE=1 target/release_debug/datafusion-bench tpch \
141154

142155
Source: `benchmarks/duckdb-bench/src/main.rs`.
143156

157+
Uses prepared data, like DataFusion. Prepare missing formats with `vx-bench prepare-data`
158+
or `cargo run --bin data-gen -- <benchmark> --formats <formats>` before profiling.
159+
144160
Supported diagnostics:
145161

146162
- `--formats parquet,vortex,vortex-compact,duckdb`;
@@ -158,7 +174,7 @@ Example:
158174
RUST_LOG=duckdb_bench=trace,vortex_duckdb=debug,info \
159175
target/release_debug/duckdb-bench tpch \
160176
--display-format gh-json --iterations 5 --hide-progress-bar \
161-
--formats <baseline-format>,<candidate-format> --queries <query> --threads 8 --reuse \
177+
--formats <baseline-format>,<candidate-format> --queries <query> --threads 8 \
162178
-o /private/tmp/<label>.jsonl
163179
```
164180

‎.agents/skills/samply/SKILL.md‎

Lines changed: 14 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -100,6 +100,15 @@ For long performance sessions, use this cadence:
100100
samply load --no-open /tmp/<label>.profile.json.gz
101101
```
102102

103+
On Linux, the summary script uses `llvm-symbolizer` for `--symbolicate`. Keep the exact
104+
recorded binary available. Check that the profile contains samples before interpreting it.
105+
If launching Samply under a CPU affinity mask yields an empty profile, start the benchmark
106+
with its desired affinity and attach Samply from a process with the full affinity mask:
107+
108+
```bash
109+
sudo samply record --save-only --rate 500 --pid <benchmark-pid> --output /tmp/<label>.profile.json.gz
110+
```
111+
103112
Then report the printed local URL.
104113

105114
4. Summarize the profile without opening Firefox Profiler. Run a small summary first and report it
@@ -175,6 +184,11 @@ Samply writes Firefox-profiler JSON, often compressed as `profile.json.gz`.
175184
`stackTable`. `samples.weight[]` is the number of collapsed samples represented by that row; use
176185
weight instead of row count when present. `samples.threadCPUDelta[]` is per-thread CPU delta in
177186
microseconds when present.
187+
- Validate CPU deltas before using CPU-weighted attribution or `profile_activity.py`.
188+
If idle threads each accumulate roughly the whole recording duration, the CPU
189+
deltas are unsuitable. Use `--weight-mode samples`, select execution workers,
190+
and compare with separately recorded perf task-clock. Timestamps may be stored
191+
as absolute `samples.time` values or cumulative `samples.timeDeltas`.
178192
- `stackTable` is a linked list: `stackTable.frame[i]` is the current frame and
179193
`stackTable.prefix[i]` points to the caller stack. Follow prefixes to `null` and reverse to get
180194
root-to-leaf order.

‎.agents/skills/samply/scripts/profile_activity.py‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,6 @@
11
#!/usr/bin/env python3
2+
# SPDX-License-Identifier: Apache-2.0
3+
# SPDX-FileCopyrightText: Copyright the Vortex contributors
24
"""Summarize thread activity over time in a Samply / Firefox profiler JSON file."""
35

46
from __future__ import annotations
@@ -14,6 +16,8 @@
1416

1517

1618
def sample_times(samples: dict[str, Any]) -> list[float]:
19+
if isinstance(samples.get("time"), list):
20+
return samples["time"]
1721
total = 0.0
1822
times = []
1923
for delta in samples.get("timeDeltas") or []:

‎.agents/skills/samply/scripts/profile_inverted_tree.py‎

Lines changed: 18 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -1,10 +1,13 @@
11
#!/usr/bin/env python3
2+
# SPDX-License-Identifier: Apache-2.0
3+
# SPDX-FileCopyrightText: Copyright the Vortex contributors
24
"""Print an inverted call tree from a Samply / Firefox profiler JSON file."""
35

46
from __future__ import annotations
57

68
import argparse
79
import collections
10+
import platform
811
import re
912
from collections.abc import Callable
1013
from pathlib import Path
@@ -20,6 +23,8 @@ def __init__(self) -> None:
2023

2124

2225
def sample_times(samples: dict[str, Any]) -> list[float]:
26+
if isinstance(samples.get("time"), list):
27+
return samples["time"]
2328
total = 0.0
2429
times = []
2530
for delta in samples.get("timeDeltas") or []:
@@ -37,13 +42,13 @@ def thread_matches(thread: dict[str, Any], thread_re: re.Pattern[str] | None, ti
3742
return bool(thread_re.search(str(thread.get("name", ""))) or thread_re.search(str(thread.get("tid", ""))))
3843

3944

40-
def frame_labels(thread: dict[str, Any], symbols: dict[str, str]) -> Callable[[int], str]:
45+
def frame_labels(thread: dict[str, Any], symbols: dict[str, str], symbol_lib: str | None) -> Callable[[int], str]:
4146
cache: dict[int, str] = {}
4247

4348
def label(frame_index: int) -> str:
4449
cached = cache.get(frame_index)
4550
if cached is None:
46-
cached = ps.frame_label(thread, frame_index, symbols)
51+
cached = ps.frame_label(thread, frame_index, symbols, symbol_lib)
4752
cache[frame_index] = cached
4853
return cached
4954

@@ -93,7 +98,7 @@ def build_tree(
9398
samples = thread.get("samples") or {}
9499
stacks = samples.get("stack") or []
95100
times = sample_times(samples)
96-
label = frame_labels(thread, symbols)
101+
label = frame_labels(thread, symbols, (args.symbol_lib or args.binary.name) if args.symbolicate else None)
97102
thread_weight = 0
98103
thread_samples = 0
99104

@@ -151,7 +156,11 @@ def main() -> int:
151156
parser.add_argument("--top", type=int, default=8, help="Max children to print at each tree level")
152157
parser.add_argument("--depth", type=int, default=12, help="Max inverted tree depth to print")
153158
parser.add_argument("--binary", type=Path, help="Candidate binary for symbolication")
154-
parser.add_argument("--symbolicate", action="store_true", help="Use atos to symbolicate raw app offsets")
159+
parser.add_argument(
160+
"--symbolicate",
161+
action="store_true",
162+
help="Symbolicate raw app offsets with llvm-symbolizer on Linux or atos on macOS",
163+
)
155164
parser.add_argument(
156165
"--symbol-lib",
157166
help="Library name to symbolicate; defaults to the basename of --binary",
@@ -166,7 +175,11 @@ def main() -> int:
166175
parser.error("--symbolicate requires --binary")
167176
symbol_lib = args.symbol_lib or args.binary.name
168177
addresses = ps.collect_symbol_addresses(profile, symbol_lib)
169-
symbols = ps.atos_symbol_map(args.binary, addresses, int(str(args.load_address), 0))
178+
symbols = (
179+
ps.llvm_symbol_map(args.binary, addresses)
180+
if platform.system() == "Linux"
181+
else ps.atos_symbol_map(args.binary, addresses, int(str(args.load_address), 0))
182+
)
170183

171184
root, matched_threads = build_tree(profile, symbols, args)
172185
unit = "cpu_us" if args.weight_mode == "cpu" else "samples"

‎.agents/skills/samply/scripts/profile_summary.py‎

Lines changed: 45 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,6 @@
11
#!/usr/bin/env python3
2+
# SPDX-License-Identifier: Apache-2.0
3+
# SPDX-FileCopyrightText: Copyright the Vortex contributors
24
"""Summarize Samply / Firefox profiler JSON without opening the UI."""
35

46
from __future__ import annotations
@@ -8,6 +10,7 @@
810
import gzip
911
import json
1012
import os
13+
import platform
1114
import re
1215
import subprocess
1316
from pathlib import Path
@@ -51,7 +54,12 @@ def resource_name(thread: dict[str, Any], resource_index: object) -> str:
5154
ADDRESS_RE = re.compile(r"^0x[0-9a-fA-F]+$")
5255

5356

54-
def frame_label(thread: dict[str, Any], frame_index: int, symbols: dict[str, str] | None = None) -> str:
57+
def frame_label(
58+
thread: dict[str, Any],
59+
frame_index: int,
60+
symbols: dict[str, str] | None = None,
61+
symbol_lib: str | None = None,
62+
) -> str:
5563
frame_table = thread.get("frameTable") or {}
5664
func_table = thread.get("funcTable") or {}
5765
func_index = value_at(frame_table, "func", frame_index)
@@ -63,7 +71,8 @@ def frame_label(thread: dict[str, Any], frame_index: int, symbols: dict[str, str
6371
file_name = decode_string(thread, value_at(func_table, "fileName", func_index))
6472
line = value_at(func_table, "lineNumber", func_index)
6573

66-
label = symbols.get(name, name) if symbols else name
74+
matching_lib = symbol_lib is None or os.path.basename(resource) == symbol_lib
75+
label = symbols.get(name, name) if symbols and matching_lib else name
6776
label = label or f"<func:{func_index}>"
6877
if resource and resource not in label:
6978
label = f"{label} [{resource}]"
@@ -113,6 +122,7 @@ def summarize_thread(
113122
thread: dict[str, Any],
114123
symbols: dict[str, str] | None = None,
115124
weight_mode: str = "samples",
125+
symbol_lib: str | None = None,
116126
) -> dict[str, Any]:
117127
samples = thread.get("samples") or {}
118128
stacks = samples.get("stack") or []
@@ -127,7 +137,7 @@ def summarize_thread(
127137
def label(frame_index: int) -> str:
128138
cached = label_cache.get(frame_index)
129139
if cached is None:
130-
cached = frame_label(thread, frame_index, symbols)
140+
cached = frame_label(thread, frame_index, symbols, symbol_lib)
131141
label_cache[frame_index] = cached
132142
return cached
133143

@@ -238,6 +248,24 @@ def atos_symbol_map(binary: Path, addresses: list[str], load_address: int) -> di
238248
return {raw: symbol.strip() for raw, symbol in zip(addresses, symbols) if symbol.strip() and symbol.strip() != raw}
239249

240250

251+
def llvm_symbol_map(binary: Path, addresses: list[str]) -> dict[str, str]:
252+
if not addresses:
253+
return {}
254+
result = subprocess.run(
255+
["llvm-symbolizer", f"--obj={binary}", "--no-inlines", "--demangle"],
256+
input="\n".join(addresses) + "\n",
257+
check=True,
258+
capture_output=True,
259+
text=True,
260+
)
261+
blocks = result.stdout.strip().split("\n\n")
262+
return {
263+
address: block.splitlines()[0]
264+
for address, block in zip(addresses, blocks)
265+
if block.splitlines() and block.splitlines()[0] != "??"
266+
}
267+
268+
241269
def main() -> int:
242270
parser = argparse.ArgumentParser(description=__doc__)
243271
parser.add_argument("profile", type=Path)
@@ -247,7 +275,7 @@ def main() -> int:
247275
parser.add_argument("--stack-depth", type=int, default=14, help="Frames to print from the leaf side")
248276
parser.add_argument("--libs", action="store_true", help="Print profile library metadata")
249277
parser.add_argument("--binary", type=Path, help="Print dwarfdump UUID for a candidate binary")
250-
parser.add_argument("--symbolicate", action="store_true", help="Use atos to symbolicate raw app offsets")
278+
parser.add_argument("--symbolicate", action="store_true", help="Use llvm-symbolizer on Linux or atos on macOS")
251279
parser.add_argument(
252280
"--symbol-lib",
253281
help="Library name to symbolicate; defaults to the basename of --binary",
@@ -275,21 +303,30 @@ def main() -> int:
275303
print(binary_uuid(args.binary))
276304

277305
symbols: dict[str, str] = {}
306+
symbol_lib = None
278307
if args.symbolicate:
279308
if not args.binary:
280309
parser.error("--symbolicate requires --binary")
281310
symbol_lib = args.symbol_lib or args.binary.name
282311
addresses = collect_symbol_addresses(profile, symbol_lib)
283-
symbols = atos_symbol_map(args.binary, addresses, int(str(args.load_address), 0))
312+
if platform.system() == "Linux":
313+
symbols = llvm_symbol_map(args.binary, addresses)
314+
else:
315+
symbols = atos_symbol_map(args.binary, addresses, int(str(args.load_address), 0))
284316
print(f"Symbolicated {len(symbols)} / {len(addresses)} raw addresses from {symbol_lib}")
285317

286318
if args.libs:
287319
print_libs(profile)
288320

289-
summaries = [summarize_thread(thread, symbols, args.weight_mode) for thread in profile.get("threads") or []]
290-
summaries.sort(key=lambda s: (s["cpu_ms"], s["weight"]), reverse=True)
321+
summaries = [
322+
summarize_thread(thread, symbols, args.weight_mode, symbol_lib) for thread in profile.get("threads") or []
323+
]
324+
summaries.sort(
325+
key=lambda s: (s["weight"], s["cpu_ms"]) if args.weight_mode == "samples" else (s["cpu_ms"], s["weight"]),
326+
reverse=True,
327+
)
291328

292-
print("\nThreads by CPU:")
329+
print("\nThreads by sample weight:" if args.weight_mode == "samples" else "\nThreads by CPU:")
293330
for summary in summaries[: max(args.threads, 20)]:
294331
print(
295332
f" cpu_ms={summary['cpu_ms']:10.3f} weight={summary['weight']:8d} "

‎benchmarks/datafusion-bench/src/lib.rs‎

Lines changed: 8 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -4,6 +4,7 @@
44
pub mod metrics;
55
pub mod tracer;
66

7+
use std::num::NonZeroUsize;
78
use std::sync::Arc;
89

910
use datafusion::datasource::file_format::FileFormat;
@@ -27,8 +28,9 @@ use vortex_datafusion::VortexFormat;
2728
use vortex_datafusion::VortexFormatFactory;
2829
use vortex_datafusion::VortexTableOptions;
2930

31+
/// Creates a benchmark session with the requested execution parallelism.
3032
#[expect(clippy::expect_used)]
31-
pub fn get_session_context() -> SessionContext {
33+
pub fn get_session_context(threads: Option<NonZeroUsize>) -> SessionContext {
3234
let mut rt_builder = RuntimeEnvBuilder::new();
3335

3436
rt_builder = rt_builder.with_cache_manager(CacheManagerConfig::default());
@@ -39,8 +41,12 @@ pub fn get_session_context() -> SessionContext {
3941

4042
let factory = VortexFormatFactory::new().with_options(vortex_table_options());
4143

44+
let mut config = SessionConfig::from_env().expect("shouldn't fail");
45+
if let Some(threads) = threads {
46+
config = config.with_target_partitions(threads.get());
47+
}
4248
let mut session_state_builder = SessionStateBuilder::new()
43-
.with_config(SessionConfig::from_env().expect("shouldn't fail"))
49+
.with_config(config)
4450
.with_runtime_env(rt)
4551
.with_default_features();
4652

‎benchmarks/datafusion-bench/src/main.rs‎

Lines changed: 13 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,7 @@
11
// SPDX-License-Identifier: Apache-2.0
22
// SPDX-FileCopyrightText: Copyright the Vortex contributors
33

4+
use std::num::NonZeroUsize;
45
use std::path::PathBuf;
56
use std::sync::Arc;
67
use std::time::Instant;
@@ -62,8 +63,9 @@ struct Args {
6263
#[arg(short, long, default_value_t = 5)]
6364
iterations: usize,
6465

66+
/// Number of Tokio workers and DataFusion execution partitions.
6567
#[arg(short, long)]
66-
threads: Option<usize>,
68+
threads: Option<NonZeroUsize>,
6769

6870
#[arg(short, long)]
6971
verbose: bool,
@@ -128,9 +130,16 @@ struct Args {
128130
options: Vec<Opt>,
129131
}
130132

131-
#[tokio::main]
132-
async fn main() -> anyhow::Result<()> {
133+
fn main() -> anyhow::Result<()> {
133134
let args = Args::parse();
135+
let mut runtime = tokio::runtime::Builder::new_multi_thread();
136+
if let Some(threads) = args.threads {
137+
runtime.worker_threads(threads.get());
138+
}
139+
runtime.enable_all().build()?.block_on(run(args))
140+
}
141+
142+
async fn run(args: Args) -> anyhow::Result<()> {
134143
let opts = Opts::from(args.options);
135144

136145
set_join_set_tracer(get_static_tracer())?;
@@ -187,7 +196,7 @@ async fn main() -> anyhow::Result<()> {
187196
|format| {
188197
let benchmark = &*benchmark;
189198
async move {
190-
let session = datafusion_bench::get_session_context();
199+
let session = datafusion_bench::get_session_context(args.threads);
191200
for sql in benchmark.engine_init_sql(Engine::DataFusion) {
192201
session.sql(&sql).await?.collect().await?;
193202
}

0 commit comments

Comments
 (0)