Repository navigation
Epic: Time series data support #10171
Copy link
Copy link
Open
0 / 10 of 1 issue completedLabels
epicPublic roadmap umbrella for a major initiative, with work tracked in sub-issues.Public roadmap umbrella for a major initiative, with work tracked in sub-issues.
Description
Summary: support time series data, i.e. tables sorted by a timestamp column with high-volume appends, range queries over time, and time-bucketed aggregations.
Sortedness
- Record a per-file sort key in the footer and fill
is_sorted/is_strict_sortedin file stats. - Binary search or zone seeking on the sort key for time-range predicates.
- Reverse scan for newest-first queries.
- Infer DataFusion
output_orderingfrom the file sort key instead of requiringWITH ORDER, and support ordering on the v2 table. - Let the compressor use
IsSortedto pick Sequence or Delta without trial encodes.
Aggregate pushdown
Filter and limit pushdown
- Allow filter and limit in the same scan and short-circuit once the limit is met.
- Push limit from DataFusion when a filter is present.
- Push TopK and join dynamic filters into zone pruning, matching DuckDB.
- Push Interval and Duration literals and
now() - intervalstyle predicates from DataFusion.
Better timestamp and other types
- Add Duration and Interval dtypes and fix the Arrow mappings that currently panic.
- Cast between timestamp units.
- Coerce time units in comparison and pruning instead of erroring.
- Finish DateTimeParts comparison kernels so sub-day range filters do not fall back to a full decode.
New scalar functions
-
date_trunc,date_bin,date_part/extract, timestamp plus or minus interval. - Push these down from DataFusion as vortex-expr.
Pruning extensions
- Falsifiers for functions of the time column, starting with
date_trunc/date_bin. - Multi-column predicates such as interval overlap.
- Wire the existing Bloom filter aggregate into the default writer for tag columns.
Adding delta
- Enable Delta in an edition and fix its cast rule.
- Delta-of-delta for near-regular timestamps.
- Re-apply Delta to monotone children like RLE indices and string offsets.
Benchmarks
- Sorted ClickBench: hits sorted by
EventTime, same queries. - TSBS DevOps or IoT style workload: last-point, time-range, group-by-bucket and newest-N queries.
- Compression-ratio suite on regular and jittered timestamps, counters and sensor doubles against Parquet, Gorilla and Pco.
- Ingest benchmark: streaming append throughput and time to first readable row.
Reactions are currently unavailable
Metadata
Metadata
Assignees
Labels
epicPublic roadmap umbrella for a major initiative, with work tracked in sub-issues.Public roadmap umbrella for a major initiative, with work tracked in sub-issues.
Activity
[-]epic: time series data support[/-][+]Epic: Time series data support[/+]DataFusioninto vortex. #10221joseph-isaacs commentedon Oct 3, 2026
DateTimeParts always splits timestamps at fixed boundaries: days, seconds of day, subseconds. Which split is best depends on the data. We should choose the split divisors per chunk at compression time (a mixed-radix split
v → (v / W1, (v % W1) / W2, v % W2)) instead of hard-coding 86400 s and 1 s. Below are BtrBlocks sizes for real data, measured in 131,072-row chunks (~1 MiB of i64). "Computed" means the best of 1 s, 1 m, 1 h, 1 d and pairs of them, chosen per chunk:[1h, 1s][1s])The sorted case is the typical time-series layout. There, splitting off days turns a monotonic column into a sawtooth, while dividing out the all-zero subseconds keeps it monotonic. The divisor has to come from a better estimate than a naive 16×64 sample: on sorted taxi, picking from that sample saved only 25% instead of 41%, and on BTC it was 3% worse than today. "No split" must also stay a candidate.
Generated by Claude Code