Repository navigation
Commit cf0104e
committed
Zstd: append values in bulk and harden the frame metadata reads
`append_to_builder` walked the decompressed frames one value at a time,
appending each through a single-value call and interleaving null runs by hand.
It now derives the value byte range from the slice's own metadata and hands the
whole region to `append_value_slices`, which sizes the offsets, byte storage and
validity once — one offset store plus one `memcpy` per value.
Adds a `VarBinViewBuilder` path as well: the frames already hold the values
contiguously, so the views can reference them in place instead of going through
the canonical array and rewriting every view a second time to rebase its buffer
index. `reconstruct_views` therefore takes the buffer index the views should
start at.
Frame metadata comes straight off disk, so the arithmetic it drives is now
checked and surfaced as errors rather than panics or wrapped lengths: value
counts, frame sizes, length prefixes and the offsets they walk to. The missing
`n_values` fallback no longer reads a byte count as a value count for
variable-width values, where that mis-attributes values to frames — it is
accepted only for the single-frame case that is still recoverable. Decompression
also writes through `WriteBuf` into uninitialized spare capacity rather than a
`&mut [u8]` over memory nothing has written yet.
Signed-off-by: Robert Kruszewski <robert@spiraldb.com>1 parent defd107 commit cf0104e
5 files changed
Lines changed: 669 additions & 99 deletions
File tree
- encodings/zstd
- src
- vortex-cuda/src/kernel/encodings
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
25 | 25 | | |
26 | 26 | | |
27 | 27 | | |
| 28 | + | |
28 | 29 | | |
29 | 30 | | |
30 | 31 | | |
| |||
0 commit comments