Check new-writer files against each edition's minimum reader in vortex-compat

[joseph-isaacs]

Summary

The existing vortex-compat suite proves the current reader can read files written by older releases. Nothing checks the other direction, which is the promise a frozen edition makes: a file written with edition E must be readable by every release at or above E's recorded min_library_version. This PR adds that check, a seeded random sweep with read-path queries, and the results of running them for every frozen core edition against a ladder of released readers.

The survey shows the recorded minimum versions of the three pre-2026 core editions cannot be honored by today's writer. The wire changes responsible all shipped in 0.37.0 to 0.84.0, before the frozen records were written (0.86.0, #9660), so the records retroactively promise compatibility that was already gone. The first release that reads every file and answers every query identically for every edition is 0.85.0. The two 2026 editions hold up completely: zero differences across 1,600 files and 5,431 queries per reader.

Changes

• vortex-compat generate --edition <id> writes the curated fixtures with a chosen frozen edition enabled. Fixtures the edition forbids are skipped.

• vortex-compat sweep --edition <id> --seeds N writes, per seed: a random canonical array through the flat layout; a canonicalized random slice of it; the array through the default and compact compressor pipelines with randomised row block size and data block target; columns of random base arrays (integers, floats, strings, booleans, timestamps with units and timezones, decimals, sequences) through random encoder stacks (FoR, ZigZag, ALP, ALPRD, Pco, Zstd, RLE, Sparse, Dict, RunEnd, FSST, DateTimeParts, DecimalByteParts); and random dict, constant and run-end arrays. Every eighth seed is eight times longer. Deterministic in the seed; writer panics and edition rejections are recorded in sweep.json.

• Every written file gets a queries.json entry: comparison filters with literals taken from the data (so zone maps are actually pruned), random column projections, sorted row-index reads, and limits. The query language is deliberately small enough for every released Python binding to express.

• vortex-compat check-reader --dir <files> --arrow-dir <dumps> decodes each file and runs each query with the current reader and compares against the old reader's Arrow IPC dumps column by column. Map keys_sorted is ignored. A current-reader panic is recorded as a failure.

• vortex-compat describe prints dtype, layout IDs and array IDs per file; vortex-compat probe evaluates one f64 comparison by pushed-down scan and by plain Rust. Both for triage.

• scripts/old_reader.py is the old reader: a published vortex-data wheel reads every file and runs its queries, dumping results as Arrow IPC. Files are read in batches by a child process; an abort costs one process and is recorded.

  cargo run -p vortex-compat -- sweep --output /tmp/sw --seeds 200 --edition core2025.10.0
  uv run --with pyarrow --with typing_extensions --with "substrait==0.23.0" --with vortex-data==0.54.0 \
    vortex-test/compat-gen/scripts/old_reader.py /tmp/sw /tmp/dumps
  cargo run -p vortex-compat -- check-reader --dir /tmp/sw --arrow-dir /tmp/dumps

  Wheels before 0.84 need the substrait==0.23.0 pin. The 0.36.0 and 0.40.0 wheels do not compile in vortex-pco, so core2025.06.0 cannot be measured at 0.40.0 with a wheel. Wheels before 0.55 have no limit parameter, so limit queries fail on them for that reason alone.

Sanity check

An injected break (a version byte prepended to the fastlanes.for metadata, stripped by the matching deserializer) passes the existing HEAD-only check 30/30 but is caught against 0.85.0 with failed to decode Protobuf message: invalid tag value: 0. The break is not in this PR.

Results: random sweep with queries (200 seeds per edition, up to 8 files per seed)

Edition written	Reader	Files passed / written	Queries passed
core2026.08.3	0.85.0, 0.87.0, 0.88.0	1600 / 1600	5431 / 5431
core2026.08.0	0.84.0, 0.85.0, 0.88.0	1489 / 1489	5099 / 5099
core2025.10.0	0.85.0	967 / 967	3286 / 3286
core2025.10.0	0.80.0	943 / 967	3214
core2025.10.0	0.70.0	914 / 967	3134
core2025.10.0	0.62.0	906 / 967	3111
core2025.10.0	0.54.0 (recorded minimum)	855 / 967	2049
core2025.06.0	0.85.0	874 / 874	2972 / 2972
core2025.06.0	0.54.0	779 / 874	1869
core2025.05.0	0.85.0	786 / 786	2580 / 2580
core2025.05.0	0.54.0	700 / 786	1596
core2025.05.0	0.36.0 (recorded minimum)	627 / 786	1441

Under every pre-2026 edition, every compressed-pipeline file is rejected at write time because the zoned layout emits vortex.max, vortex.bounded_max and vortex.null_count aggregates no old edition permits, so those rows hold flat-layout files only and the default write strategy is unusable under an old edition as things stand. The curated fixtures (30 files) give the same picture: 26/29 on 0.54.0 for core2025.10.0, 16/23 on 0.36.0 for core2025.05.0, and 100% on 0.85.0.

Wire changes under unchanged IDs

Each was traced to its commit and first release, and the old reader's error string was confirmed in that release's own crate source.

Component	What the writer now emits	Old reader result	Change	First release	Breaks editions with minimum below
vortex.sparse	string values	Cannot convert DType utf8? into PType	#3848	0.42.0	2025.05, 2025.06
vortex.fsst	3 buffers	Expected 2 buffers, got 3	#6325 (changelog/break)	0.59.0	2025.05, 2025.06, 2025.10
vortex.pco	pco format version 4	format version (4) exceeds max supported (3), then process abort. pco 1.x cannot write the old version	#6354	0.59.0	2025.06, 2025.10
vortex.dict	signed integer codes	expected type: unsigned int but instead got i32	#6336	0.59.0	2025.05, 2025.06, 2025.10
vortex.runend	string values	RunEnd array can only have Bool or Primitive values	#6788	0.62.0	2025.05, 2025.06, 2025.10
vortex.struct layout	validity child for nullable structs	Struct layout has N+1 children, but dtype has N fields	#8638	0.78.0	2025.05, 2025.06, 2025.10
vortex.constant	fixed-size-list scalars as ListValue	expected List dtype for ListValue	#8696	0.78.0	2025.05, 2025.06, 2025.10
vortex.constant	struct scalars as ListValue	expected List dtype for ListValue	#8838	0.80.0	2025.05, 2025.06, 2025.10
Map dtype	any Map-typed array the edition permits, e.g. vortex.constant	Unknown DType variant / field DType must be valid, process abort	#9107	0.84.0	2025.05, 2025.06, 2025.10

The Map row exposes a gap in the model: editions govern array, layout, extension-dtype and aggregate IDs, but not core DType variants, so a new logical type reaches old readers through any permitted encoding.

The query comparison found no divergence in filter pruning, projection, random access or limits against any reader from 0.84.0 on. Against older readers the query failures are the same wire changes above, reached through a different read path, plus reader-side bugs below.

Reader-side bugs the sweep also surfaced (not writer regressions)

• 0.88.0 aborts on dict-of-bool (Failed to downcast). 0.87.0 and HEAD read the file. Fixed in HEAD by Canonical dict execute handles edge cases in Decimal/Bool logic #10098, merged the day 0.88.0 was tagged, so the release carries it.
• HEAD debug assertion on nested nullable structs via the compressed pipeline: ArrayStreamAdapter sees an inner struct read back as non-nullable while the file dtype says nullable. Release builds skip the assertion and produce the right Arrow schema, so this is a latent invariant violation on develop. Reproduce with the previous sweep generator at seed 640.
• 0.36.0: nullable-struct child indexing (fix: struct array must skip its validity child #3445, fixed 0.37.0), null scalars, wide decimals. 0.54.0: zero-width fixed-size list abort. 0.62.0: fixed-size-list Arrow conversion and an is_zero_copy_to_list assertion. 0.70.0: same ListValue limits as above. All fixed later.
• HEAD API panics on legal input, seen while generating: Sparse::encode on bool and string arrays, dict_encode on bool, Pco::from_primitive on 8-bit types (unreachable!), DateTimeParts and DecimalByteParts on out-of-range values. The sweep falls back to the canonical column when an encoder panics.

What this suggests

Either the three pre-2026 records should have min_library_version raised to 0.85.0, or the writer needs legacy serializers behind new IDs for the components above when an old edition is selected. pco has no writer-side option at all. DType variants need to be edition members or the writer must reject dtypes an edition's readers cannot parse, and the default strategy should restrict zone-map aggregates to the permitted set rather than failing. The check-editions freeze guard currently forbids editing a frozen record, so raising the minimums needs a deliberate exception.

Not done

• No CI job yet. The natural shape is a matrix over vortex/editions/core/*.toml, reading each record's min_library_version, running generate and sweep with that edition and old_reader.py with that wheel.
• check-reader compares against the current reader, not an independent oracle. Where results differed, an independent pyarrow count was used to decide; every such case turned out to be a harness bug (serde_json's inexact default float parsing, now fixed with float_roundtrip), not a reader bug.
• The reverse direction with random data (old wheel writes, HEAD reads) is not covered; the curated fixtures remain the only old-writer coverage.
• Checks run: cargo build -p vortex-compat, uvx ruff format and uvx ruff check on the script. Not run: clippy, rustfmt, nextest.

🤖 Generated with Claude Code

https://claude.ai/code/session_01BS1uCYLpjJK6wBjvdMDgBV

[codspeed]

Merging this PR will degrade performance by 11.52%

⚠️ Unknown Walltime execution environment detected

Using the Walltime instrument on standard Hosted Runners will lead to inconsistent data.

For the most accurate results, we recommend using CodSpeed Macro Runners: bare-metal machines fine-tuned for performance measurement consistency.

❌ 1 regressed benchmark
✅ 2174 untouched benchmarks
⏩ 534 skipped benchmarks¹

Warning

Please fix the performance issues or acknowledge them on CodSpeed.

Performance Changes

	Mode	Benchmark	BASE	HEAD	Efficiency
❌	WallTime	bitpack_blocked_compress_avx2	6.7 µs	7.6 µs	-11.52%

Tip

Investigate this regression by commenting @codspeedbot fix this regression on this PR, or directly use the CodSpeed MCP with your agent.

---

_{Comparing ji/nifty-cerf-m02guh (0a2f7a5) with develop (881b867)}

Footnotes

1. 534 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports. ↩