DictStreamState::encode loops forever on a value that does not fit an empty dictionary

What happened?

The dict layout writer never finishes when a string value is longer than max_bytes - 16, the most an empty dictionary can hold. It spins on one CPU and memory grows until the process is killed.

BytesDictBuilder::encode_value correctly returns None for such a value:

if self.dict_bytes() + additional_bytes > self.max_dict_bytes {
    return None;
}

The caller does not handle that case. start_encoding / encode_chunk encode 0 rows, remainder returns the whole chunk, and DictStreamState::encode resets the encoder and tries the same chunk again. The loop never makes progress, and every pass pushes an empty codes chunk and an empty values chunk into res, so res grows forever.

With the default DictLayoutConstraints (1 MiB), any string or binary column that DictStrategy puts on the dict path hangs on a value over 1,048,560 bytes. A value of exactly 1 MiB is enough. We hit this in production on real string data.

Expected: DictStreamState::encode returns. Actual: it never returns.

Steps to reproduce

Add to the tests module in vortex-layout/src/layouts/dict/writer.rs. With a 64-byte dictionary, a 48-byte value passes and a 49-byte value hangs.

/// Encodes one string of `value_len` bytes under a 64-byte dictionary limit and reports
/// whether encoding finished within the timeout.
fn dict_encode_finishes(value_len: usize) -> bool {
    use std::sync::mpsc;
    use std::time::Duration;

    use super::DictChunkLabeler;
    use super::DictStreamState;

    let (done_tx, done_rx) = mpsc::channel();
    std::thread::spawn(move || {
        let mut state = DictStreamState {
            encoder: None,
            constraints: DictConstraints {
                max_bytes: 64,
                max_len: 100,
            },
        };
        let value = "x".repeat(value_len);
        let chunk = VarBinArray::from(vec![value.as_str()]).into_array();
        let mut labeler = DictChunkLabeler::new(SequenceId::root().advance());
        let encoded = state.encode(&mut labeler, chunk, &mut SESSION.create_execution_ctx());
        done_tx.send(encoded.is_ok()).ok();
    });
    done_rx.recv_timeout(Duration::from_secs(2)) == Ok(true)
}

/// A dictionary entry costs its bytes plus one 16-byte `BinaryView`, so 48 bytes is the
/// longest value that fits in a 64-byte dictionary.
#[test]
fn test_encode_value_that_fills_an_empty_dictionary() {
    assert!(dict_encode_finishes(48));
}

#[test]
fn test_encode_value_longer_than_an_empty_dictionary() {
    assert!(
        dict_encode_finishes(49),
        "DictStreamState::encode keeps retrying a chunk it encoded 0 rows of"
    );
}

$ cargo test -p vortex-layout --lib test_encode_value
test layouts::dict::writer::tests::test_encode_value_longer_than_an_empty_dictionary ... FAILED
test layouts::dict::writer::tests::test_encode_value_that_fills_an_empty_dictionary ... ok

The thread only exists for the timeout. In those 2 s the test process grew to about 230 MB RSS (debug build).

Environment

Vortex version: develop @ e0c00d540 (2026-10-08)
Rust 1.98.0 (repo toolchain)
OS: macOS 26, arm64
Python/Java: n/a

Additional context

Related: BytesDictBuilder panics instead of enforcing DictConstraints::max_bytes above 4 GiB #9687 (fixed in fix(dict): cap max_dict_bytes at the u32 range BinaryView addresses #9690) hit the same max_dict_bytes check in encode_value, but panicked before reaching it. Here the check works and the caller loops on its None.
Found and reproduced with help from Claude Code.

@claude

@claude make a fix

@robert3005

Claude finished @robert3005's task in 4m 10s —— View job • claude/issue-10397-20261008-2131 • Create PR ➔

Fixed `DictStreamState::encode` infinite loop

Read issue and reproduce the root cause
Locate the no-progress condition in the dict builders
Apply fix: always accept the first dictionary entry
Add regression tests
Commit and push

Root cause: BytesDictBuilder::encode_value returns None for a value too large to fit an empty dictionary (its bytes plus one 16-byte BinaryView exceed max_bytes). The encoder then encodes 0 rows, remainder hands back the whole chunk, and DictStreamState::encode resets the encoder and retries the identical chunk forever — pushing an empty codes/values chunk each pass, so memory grows unbounded.

Fix: guarantee forward progress by always admitting the first entry into an empty dictionary, even when it exceeds the byte or length budget. An oversized value is then written as its own single-entry dictionary, and the surrounding values still encode normally. I applied the same guard to PrimitiveDictBuilder, which had the identical no-progress risk when max_bytes is smaller than a single element.

Changes:

vortex-array/src/builders/dict/bytes.rs — encode_value / encode_null skip the budget checks for the first entry.
vortex-array/src/builders/dict/primitive.rs — same guard in encode_value / encode_null.
vortex-layout/src/layouts/dict/writer.rs — two regression tests: one asserts a 49-byte value under a 64-byte dictionary encodes and returns; one asserts an oversized value in the middle of a chunk is split into its own dictionary while neighbours encode normally. These assert termination directly (no thread/timeout needed now that the loop makes progress).

Note on verification: cargo is blocked by this environment's sandbox, so I could not run cargo test -p vortex-layout / clippy here. Please run them in CI:

cargo test -p vortex-layout --lib layouts::dict::writer::tests
cargo nextest run -p vortex-array --lib builders::dict
cargo clippy -p vortex-array -p vortex-layout --all-targets --all-features -- -D warnings

· claude/issue-10397-20261008-2131

Navigation Menu

Repository navigation

DictStreamState::encode loops forever on a value that does not fit an empty dictionary #10397

What happened?

Steps to reproduce

Environment

Additional context

Fixed `DictStreamState::encode` infinite loop

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

DictStreamState::encode loops forever on a value that does not fit an empty dictionary #10397

Description

What happened?

Steps to reproduce

Environment

Additional context

Activity

robert3005 commented on Oct 8, 2026

vortex-claude commented on Oct 8, 2026

Fixed DictStreamState::encode infinite loop

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions

Fixed `DictStreamState::encode` infinite loop