Repository navigation
Add the JSONBench benchmark suite - #10354
joseph-isaacs wants to merge 3 commits into
Conversation
ClickHouse's JSONBench runs five analytical queries over Bluesky events
stored as semi-structured JSON. Add it to vortex-bench as `jsonbench`,
with the same events stored three ways: Parquet with the event as a
JSON string, Parquet with a shredded Parquet Variant column, and Vortex
with a shredded Vortex Variant column converted from the Parquet Variant
files.
Queries read JSON paths through `{str:a.b}` and `{i64:a.b}` placeholders
that each engine expands for each format, through the new
`Benchmark::query_for`. DataFusion gets datafusion-functions-json for
the JSON strings. Data generation and the DuckDB runner now run a
suite's own `prepare_format` before the generic Parquet to Vortex
conversion, so JSONBench writes its Vortex files from the Parquet
Variant files.
The suite is not in the CI benchmark matrix. DataFusion over the Variant
formats and DuckDB over Vortex need Variant support in those engines,
which lands separately.
Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>
Co-Authored-By: Claude <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01P9G2RwAZnBNHUackdtcjeq
Store the data directory directly instead of re-deriving it from the URL, run Variant Parquet preparation once and off the async runtime when preparing Vortex, make placeholder expansion a free function, and build the raw file URL from its file name. Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01HXvxbPjYxoaTEqBgPLXbad
Merging this PR will not alter performance
|
Codecov Report✅ All modified and coverable lines are covered by tests. ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk> Co-Authored-By: Claude <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01P9G2RwAZnBNHUackdtcjeq
Summary
Adds ClickHouse's JSONBench to
vortex-benchasjsonbench. It runs five analytical queries over Bluesky events stored as semi-structured JSON. It is the first suite that exercises Variant columns, and it gives the upcoming Variant work a baseline to measure against.This PR only adds the suite. It is not in the CI benchmark matrix. Two kinds of target need engine support that will come in follow-up PRs:
parquet-variantandvortex: DataFusion has novariant_getfunction registered yet.vortex:read_vortexdoes not expose Vortex Variant columns yet.The JSON-string Parquet baseline only uses functions that already exist: datafusion-functions-json on DataFusion and
json_extract_stringon DuckDB.Changes
vortex-bench/src/jsonbench--opt scale-factor=Npicks how many files.shredding.json.parquet: the event as a JSON string.parquet-variant: the event as a shredded Parquet Variant column. This is the newFormat::ParquetVariant.vortex: the event as a shredded Vortex Variant column, converted from the Parquet Variant files.vortex-bench/sql/jsonbench.sql) read JSON paths through{str:a.b}and{i64:a.b}placeholders. The newBenchmark::query_forlets a suite rewrite its SQL for each engine and format; by default it returns the query unchanged.duckdb-benchnow run a suite'sprepare_formatbefore the generic Parquet-to-Vortex conversion, so JSONBench writes its own Vortex files. The Parquet-to-Vortex conversion now hands Variant chunks to the writer unchanged, because Variant has no array builder.datafusion-benchregisters datafusion-functions-json when running JSONBench. It is a new workspace dependency.VX_BENCH_PRINT_RESULTS=1prints each query's result, for comparing formats and engines.jsonbenchsuite and theparquet-variantformat, but neither is added to a CI preset.vortex-bench/sql/jsonbench.md.