Skip to content

Epic: Time series data support #10171

Open
0 of 1 issue completed
Open

Description

@joseph-isaacs

Summary: support time series data, i.e. tables sorted by a timestamp column with high-volume appends, range queries over time, and time-bucketed aggregations.

Sortedness

  • Record a per-file sort key in the footer and fill is_sorted / is_strict_sorted in file stats.
  • Binary search or zone seeking on the sort key for time-range predicates.
  • Reverse scan for newest-first queries.
  • Infer DataFusion output_ordering from the file sort key instead of requiring WITH ORDER, and support ordering on the v2 table.
  • Let the compressor use IsSorted to pick Sequence or Delta without trial encodes.

Aggregate pushdown

Filter and limit pushdown

  • Allow filter and limit in the same scan and short-circuit once the limit is met.
  • Push limit from DataFusion when a filter is present.
  • Push TopK and join dynamic filters into zone pruning, matching DuckDB.
  • Push Interval and Duration literals and now() - interval style predicates from DataFusion.

Better timestamp and other types

  • Add Duration and Interval dtypes and fix the Arrow mappings that currently panic.
  • Cast between timestamp units.
  • Coerce time units in comparison and pruning instead of erroring.
  • Finish DateTimeParts comparison kernels so sub-day range filters do not fall back to a full decode.

New scalar functions

  • date_trunc, date_bin, date_part / extract, timestamp plus or minus interval.
  • Push these down from DataFusion as vortex-expr.

Pruning extensions

  • Falsifiers for functions of the time column, starting with date_trunc / date_bin.
  • Multi-column predicates such as interval overlap.
  • Wire the existing Bloom filter aggregate into the default writer for tag columns.

Adding delta

  • Enable Delta in an edition and fix its cast rule.
  • Delta-of-delta for near-regular timestamps.
  • Re-apply Delta to monotone children like RLE indices and string offsets.

Benchmarks

  • Sorted ClickBench: hits sorted by EventTime, same queries.
  • TSBS DevOps or IoT style workload: last-point, time-range, group-by-bucket and newest-N queries.
  • Compression-ratio suite on regular and jittered timestamps, counters and sensor doubles against Parquet, Gorilla and Pco.
  • Ingest benchmark: streaming append throughput and time to first readable row.

Activity

  1. changed the title [-]epic: time series data support[/-] [+]Epic: Time series data support[/+] on Oct 1, 2026
  2. added
    epicPublic roadmap umbrella for a major initiative, with work tracked in sub-issues.
    on Oct 1, 2026
  3. joseph-isaacs commented on Oct 3, 2026

    @joseph-isaacs
    ContributorAuthor

    DateTimeParts always splits timestamps at fixed boundaries: days, seconds of day, subseconds. Which split is best depends on the data. We should choose the split divisors per chunk at compression time (a mixed-radix split v → (v / W1, (v % W1) / W2, v % W2)) instead of hard-coding 86400 s and 1 s. Below are BtrBlocks sizes for real data, measured in 131,072-row chunks (~1 MiB of i64). "Computed" means the best of 1 s, 1 m, 1 h, 1 d and pairs of them, chosen per chunk:

    column today best computed split change
    NYC taxi 2023-01 pickup, file order (3.07M rows, µs) 6.53 MB 4.81 MB [1h, 1s] −26%
    NYC taxi pickup, sorted by time 2.64 MB 1.57 MB (mostly [1s]) −41%
    NYC taxi dropoff, file order 6.63 MB 6.12 MB −8%
    Binance BTCUSDT trades 2024-01-01 (1.11M rows, ms, sorted) 1.52 MB 1.51 MB ~0

    The sorted case is the typical time-series layout. There, splitting off days turns a monotonic column into a sawtooth, while dividing out the all-zero subseconds keeps it monotonic. The divisor has to come from a better estimate than a naive 16×64 sample: on sorted taxi, picking from that sample saved only 25% instead of 41%, and on BTC it was 3% worse than today. "No split" must also stay a candidate.


    Generated by Claude Code

  4. self-assigned this
    on Oct 13, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

epicPublic roadmap umbrella for a major initiative, with work tracked in sub-issues.

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions