Skip to content

Dict fill_null can truncate indices of unreferenced dictionary values #10623

Open

Description

@connortsui20

What happened?

Dict::fill_null casts the matching dictionary index to the codes' native integer type without checking that it fits. A dictionary can contain unreferenced entries beyond the codes' range, so this can silently fill with the wrong value.

For nullable u8 codes [0, null] and i32 dictionary values 0..=300, fill_null(300) should return [0, 300]. The unchecked cast truncates dictionary index 300 to 44, producing [0, 44].

Steps to reproduce

With an execution context ctx:

let codes = PrimitiveArray::from_option_iter([Some(0u8), None]).into_array();
let values = PrimitiveArray::from_iter(0i32..=300).into_array();
let dict = DictArray::try_new(codes, values)?;
let filled = dict
    .into_array()
    .fill_null(300i32)?
    .execute::<PrimitiveArray>(&mut ctx)?;

Environment

Source inspected on develop at 61ff28d97069f8ab028c17d38c8ad226a8b31f7c. This is a source-only finding. The reproduction and regression tests have not been run.

Additional context

The unchecked cast is in Dict::fill_null. Signed code types have the same problem when the fill index exceeds their maximum.

Found while reviewing #10332, but the example uses ordinary primitive codes and does not require Narrow. A fix is prepared locally for that PR: check the index conversion and fall back to filling decoded values when it does not fit.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugA bug issue

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions