Skip to content

Add the JSONBench benchmark suite - #10354

Open
joseph-isaacs wants to merge 3 commits into
developfrom
ji/jsonbench-suite
Open

joseph-isaacs wants to merge 3 commits into
developfrom
ji/jsonbench-suite

Conversation

@joseph-isaacs

@joseph-isaacs joseph-isaacs commented Oct 7, 2026 •

Copy link
Copy Markdown
Contributor

Summary

Adds ClickHouse's JSONBench to vortex-bench as jsonbench. It runs five analytical queries over Bluesky events stored as semi-structured JSON. It is the first suite that exercises Variant columns, and it gives the upcoming Variant work a baseline to measure against.

This PR only adds the suite. It is not in the CI benchmark matrix. Two kinds of target need engine support that will come in follow-up PRs:

  • DataFusion over parquet-variant and vortex: DataFusion has no variant_get function registered yet.
  • DuckDB over vortex: read_vortex does not expose Vortex Variant columns yet.

The JSON-string Parquet baseline only uses functions that already exist: datafusion-functions-json on DataFusion and json_extract_string on DuckDB.

Changes

  • vortex-bench/src/jsonbench
    • Downloads the raw event files from ClickHouse's public bucket, one file per million events. --opt scale-factor=N picks how many files.
    • Infers which JSON paths to shred from the data and saves them to shredding.json.
    • Writes three formats from the same events:
      • parquet: the event as a JSON string.
      • parquet-variant: the event as a shredded Parquet Variant column. This is the new Format::ParquetVariant.
      • vortex: the event as a shredded Vortex Variant column, converted from the Parquet Variant files.
  • Queries (vortex-bench/sql/jsonbench.sql) read JSON paths through {str:a.b} and {i64:a.b} placeholders. The new Benchmark::query_for lets a suite rewrite its SQL for each engine and format; by default it returns the query unchanged.
  • Data generation and duckdb-bench now run a suite's prepare_format before the generic Parquet-to-Vortex conversion, so JSONBench writes its own Vortex files. The Parquet-to-Vortex conversion now hands Variant chunks to the writer unchanged, because Variant has no array builder.
  • datafusion-bench registers datafusion-functions-json when running JSONBench. It is a new workspace dependency.
  • VX_BENCH_PRINT_RESULTS=1 prints each query's result, for comparing formats and engines.
  • bench-orchestrator knows the jsonbench suite and the parquet-variant format, but neither is added to a CI preset.
  • Docs: vortex-bench/sql/jsonbench.md.

ClickHouse's JSONBench runs five analytical queries over Bluesky events
stored as semi-structured JSON. Add it to vortex-bench as `jsonbench`,
with the same events stored three ways: Parquet with the event as a
JSON string, Parquet with a shredded Parquet Variant column, and Vortex
with a shredded Vortex Variant column converted from the Parquet Variant
files.

Queries read JSON paths through `{str:a.b}` and `{i64:a.b}` placeholders
that each engine expands for each format, through the new
`Benchmark::query_for`. DataFusion gets datafusion-functions-json for
the JSON strings. Data generation and the DuckDB runner now run a
suite's own `prepare_format` before the generic Parquet to Vortex
conversion, so JSONBench writes its Vortex files from the Parquet
Variant files.

The suite is not in the CI benchmark matrix. DataFusion over the Variant
formats and DuckDB over Vortex need Variant support in those engines,
which lands separately.

Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>
Co-Authored-By: Claude <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01P9G2RwAZnBNHUackdtcjeq
@joseph-isaacs joseph-isaacs added the changelog/chore A trivial change label Oct 7, 2026 — with Claude
Store the data directory directly instead of re-deriving it from the URL,
run Variant Parquet preparation once and off the async runtime when
preparing Vortex, make placeholder expansion a free function, and build
the raw file URL from its file name.

Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HXvxbPjYxoaTEqBgPLXbad
@codspeed

codspeed Bot commented Oct 7, 2026 •

Copy link
Copy Markdown

Merging this PR will not alter performance

⚠️ Unknown Walltime execution environment detected

Using the Walltime instrument on standard Hosted Runners will lead to inconsistent data.

For the most accurate results, we recommend using CodSpeed Macro Runners: bare-metal machines fine-tuned for performance measurement consistency.

✅ 2111 untouched benchmarks
⏩ 518 skipped benchmarks1


Comparing ji/jsonbench-suite (1d6ff33) with develop (f234bc6)

Open in CodSpeed

Footnotes

  1. 518 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports. ↩

@codecov

codecov Bot commented Oct 7, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 83.26%. Comparing base (9eb20ba) to head (b20730a).
⚠️ Report is 1 commits behind head on develop.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>
Co-Authored-By: Claude <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01P9G2RwAZnBNHUackdtcjeq

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

changelog/chore A trivial change

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants