Skip to content

[SPARK-58766][PYTHON][TEST][FOLLOWUP] Render NaT-sentinel time64 scalars in golden files #58266

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
Spenserrrr wants to merge 1 commit into from
+59 −49
Closed

[SPARK-58766][PYTHON][TEST][FOLLOWUP] Render NaT-sentinel time64 scalars in golden files #58266

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
16 changes: 13 additions & 3 deletions python/pyspark/testing/goldenutils.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -316,13 +316,16 @@ def _scalar_str(scalar: Any) -> str:
"""
Render one PyArrow scalar for a golden cell via PyArrow's own ``str(scalar)``.

Some values are valid in Arrow yet unrenderable via ``str``:
Some values that Arrow stores are unrenderable via ``str``:
- a temporal value past Python's ``datetime`` range (e.g. date32 past year 9999)
raises ``OverflowError`` -> the marker ``temporal overflow``;
- an unsafe ``binary``->``string`` cast relabels bytes without UTF-8 validation,
so non-UTF-8 bytes raise ``UnicodeDecodeError`` -> the raw bytes, so the golden
still tracks what Arrow stored for them.
An unexpected error (non-temporal overflow, non-string decode error) propagates.
still tracks what Arrow stored for them;
- a nanosecond ``time64`` holding INT64_MIN collides with pandas' NaT sentinel and
raises ``ValueError`` -> the marker ``NaT collision``.
An unexpected error (non-temporal overflow, non-string decode error, or a
ValueError from any other value) propagates.
"""
try:
return str(scalar).replace("\x00", "\\0")
Expand All @@ -341,6 +344,13 @@ def _scalar_str(scalar: Any) -> str:
if pa.types.is_string(scalar.type) or pa.types.is_large_string(scalar.type):
return repr(scalar.as_buffer().to_pybytes())
raise
except ValueError:
# A nanosecond time64 renders through pandas (Python's ``time`` is microsecond
# resolution), and pandas reads INT64_MIN as its NaT sentinel, so it refuses
# that one value. Any other ValueError is unexpected, so re-raise.
if pa.types.is_time(scalar.type) and scalar.value == pd.NaT.value:
return "NaT collision"
raise

@classmethod
def repr_arrow_value(
Expand Down
4 changes: 2 additions & 2 deletions python/pyspark/tests/upstream/pyarrow/golden_pyarrow_array_from_pandas_type_scalar_safe.csv
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
source \ target int8 int64 float32 timestamp[us] date32[day] duration[us] time64[ns] string binary
int64:standard ERR@ArrowInvalid [0, 1, -1, 9223372036854775807, -9223372036854775808]@int64 ERR@ArrowInvalid [1970-01-01 00:00:00, 1970-01-01 00:00:00.000001, 1969-12-31 23:59:59.999999, temporal overflow, temporal overflow]@timestamp[us] ERR@ArrowNotImplementedError [0:00:00, 0:00:00.000001, -1 day, 23:59:59.999999, 106751991 days, 4:00:54.775807, -106751992 days, 19:59:05.224192]@duration[us] ERR@ValueError ERR@ArrowTypeError [b'', b'\x01', b'\xff\xff\xff\xff\xff\xff\xff\xff', b'\xff\xff\xff\xff\xff\xff\xff\x7f', b'']@binary
int64:standard ERR@ArrowInvalid [0, 1, -1, 9223372036854775807, -9223372036854775808]@int64 ERR@ArrowInvalid [1970-01-01 00:00:00, 1970-01-01 00:00:00.000001, 1969-12-31 23:59:59.999999, temporal overflow, temporal overflow]@timestamp[us] ERR@ArrowNotImplementedError [0:00:00, 0:00:00.000001, -1 day, 23:59:59.999999, 106751991 days, 4:00:54.775807, -106751992 days, 19:59:05.224192]@duration[us] [00:00:00, 00:00:00, 23:59:59.999999, 23:47:16.854775, NaT collision]@time64[ns] ERR@ArrowTypeError [b'', b'\x01', b'\xff\xff\xff\xff\xff\xff\xff\xff', b'\xff\xff\xff\xff\xff\xff\xff\x7f', b'']@binary
float64:standard ERR@ArrowInvalid ERR@ArrowInvalid [0.0, 1.5, -1.5]@float32 ERR@ArrowNotImplementedError ERR@ArrowNotImplementedError ERR@ArrowNotImplementedError ERR@ArrowNotImplementedError ERR@ArrowTypeError [b'', b'', b'']@binary
bool:standard [1, 0, 1]@int8 [1, 0, 1]@int64 [1.0, 0.0, 1.0]@float32 ERR@ArrowNotImplementedError ERR@ArrowNotImplementedError ERR@ArrowNotImplementedError ERR@ArrowNotImplementedError ERR@ArrowTypeError [b'\x01', b'', b'\x01']@binary
object:string ERR@ArrowInvalid ERR@ArrowInvalid ERR@ArrowInvalid ERR@ArrowTypeError ERR@ArrowTypeError ERR@ArrowTypeError ERR@ArrowTypeError [hello, world, ]@string [b'hello', b'world', b'']@binary
Expand All @@ -8,7 +8,7 @@ date:standard ERR@ArrowInvalid ERR@ArrowInvalid ERR@ArrowInvalid ERR@ArrowTypeEr
object:datetime ERR@ArrowInvalid ERR@ArrowInvalid ERR@ArrowInvalid [2024-06-15 18:30:00]@timestamp[us] [2024-06-15]@date32[day] ERR@ArrowTypeError ERR@ArrowTypeError ERR@ArrowTypeError ERR@ArrowTypeError
object:timedelta ERR@ArrowInvalid ERR@ArrowInvalid ERR@ArrowInvalid ERR@ArrowTypeError ERR@ArrowTypeError [1 day, 2:00:00]@duration[us] ERR@ArrowTypeError ERR@ArrowTypeError ERR@ArrowTypeError
time:standard ERR@ArrowInvalid ERR@ArrowInvalid ERR@ArrowInvalid ERR@ArrowTypeError ERR@ArrowTypeError ERR@ArrowTypeError [18:30:45]@time64[ns] ERR@ArrowTypeError ERR@ArrowTypeError
Int64:standard ERR@ArrowInvalid [0, 1, 9223372036854775807, -9223372036854775808]@int64 ERR@ArrowInvalid [1970-01-01 00:00:00, 1970-01-01 00:00:00.000001, temporal overflow, temporal overflow]@timestamp[us] ERR@ArrowNotImplementedError [0:00:00, 0:00:00.000001, 106751991 days, 4:00:54.775807, -106751992 days, 19:59:05.224192]@duration[us] ERR@ValueError ERR@ArrowTypeError [b'', b'\x01', b'\xff\xff\xff\xff\xff\xff\xff\x7f', b'']@binary
Int64:standard ERR@ArrowInvalid [0, 1, 9223372036854775807, -9223372036854775808]@int64 ERR@ArrowInvalid [1970-01-01 00:00:00, 1970-01-01 00:00:00.000001, temporal overflow, temporal overflow]@timestamp[us] ERR@ArrowNotImplementedError [0:00:00, 0:00:00.000001, 106751991 days, 4:00:54.775807, -106751992 days, 19:59:05.224192]@duration[us] [00:00:00, 00:00:00, 23:47:16.854775, NaT collision]@time64[ns] ERR@ArrowTypeError [b'', b'\x01', b'\xff\xff\xff\xff\xff\xff\xff\x7f', b'']@binary
string[pyarrow]:standard ERR@ArrowInvalid ERR@ArrowInvalid ERR@ArrowInvalid ERR@ArrowInvalid ERR@ArrowInvalid ERR@ArrowNotImplementedError ERR@ArrowNotImplementedError [hello, world]@string [b'hello', b'world']@binary
large_binary[pyarrow]:standard ERR@ArrowInvalid ERR@ArrowInvalid ERR@ArrowInvalid ERR@ArrowNotImplementedError ERR@ArrowNotImplementedError ERR@ArrowNotImplementedError ERR@ArrowNotImplementedError [hello, world]@string [b'hello', b'world']@binary
int64:overflow ERR@ArrowInvalid [300, 1]@int64 [300.0, 1.0]@float32 [1970-01-01 00:00:00.000300, 1970-01-01 00:00:00.000001]@timestamp[us] ERR@ArrowNotImplementedError [0:00:00.000300, 0:00:00.000001]@duration[us] [00:00:00, 00:00:00]@time64[ns] ERR@ArrowTypeError [b',\x01', b'\x01']@binary
Expand Down
Loading