fix: fallback character-set detection for file-like objects - #4438
Conversation
text_head() decoded file-like content with errors='ignore', silently stripping undecodable characters for non-UTF-8 streams (S3/GCS objects, API uploads). Use the same detect_file_encoding() fallback as the file-path branch when the declared encoding cannot decode the content. Regression for Unstructured-IO#4434; removes the resolved TODO. Signed-off-by: fei <204683769+feiiiiii5@users.noreply.github.com>
![cubic-dev-ai[bot]](https://avatars.githubusercontent.com/in/1082092?s=60&v=4){kind=small}
There was a problem hiding this comment.
All reported issues were addressed across 4 files
Shadow auto-approve: would not auto-approve because issues were found.
Re-trigger cubic
The file-like text_head branch reads exactly 4096 bytes and strict-decodes them, so a multi-byte character split at the read boundary raised UnicodeDecodeError even for correctly-encoded content, misdiagnosing it as a wrong-encoding case and routing it into detect_file_encoding on truncated bytes. Decode incrementally with final=False so the incomplete trailing sequence is buffered instead of raising; genuine mid-stream decode errors still fall through to character-set detection. Also un-xfail the utf-32 file-like test: the TODO it tracked is now fixed, and the strict xfail xpasses; add a regression test for a boundary-split utf-8 character.
There was a problem hiding this comment.
All reported issues were addressed across 2 files (changes from recent commits).
Shadow auto-approve: would not auto-approve because issues were found.
Re-trigger cubic
final=False buffered a trailing incomplete sequence even when the stream had been fully read, silently dropping the tail without ever running character-set detection. Detect EOF after the 4096-byte read and pass final=eof_reached to the incremental decoder, so a boundary split in a longer stream is still buffered while a genuinely truncated stream falls through to detection.
There was a problem hiding this comment.
All reported issues were addressed across 2 files (changes from recent commits).
Shadow auto-approve: would not auto-approve because issues were found.
Tip: Review your code locally with the cubic CLI to iterate faster.
Re-trigger cubic
Two review follow-ups on text_head():
- compute eof_reached only for byte streams; text streams return
before the EOF probe, so the probe no longer runs on their behalf
- regression test now asserts a truncated utf-8 tail is not silently
decoded to its ASCII prefix ("caf") but falls through to character
detection
The probe must still read one byte past the 4096-byte window, so it
stays immediately after the read instead of after the str early-return.
There was a problem hiding this comment.
0 issues found across 2 files (changes from recent commits).
Shadow auto-approve: would auto-approve. Bug fix confined to file-like object text-head decoding; applies fallback encoding detection and incremental decoder for boundary splits, with tests pinning corrected behavior.
Re-trigger cubic
cragwolfe
left a comment
cragwolfe
left a comment
There was a problem hiding this comment.
Found one merge-blocking issue:
test_unstructured/file_utils/test_filetype.py:22importsdetect_file_encodingbut never uses it. The repository’sruff check .reports F401 and exits nonzero, which fails the lint matrix and prevents downstream test jobs from running. Remove the unused import; acceptance is that Ruff passes and the file-type tests remain green.
(authored by codex)
P3-a dropped the last use of the encoding-detection import, so it fails ruff F401 in the lint matrix.
|
Addressed in commit a7f3405: removed the unused detect_file_encoding import from test_unstructured/file_utils/test_filetype.py. Ruff 0.15.2 check passes on the file, and the file-type tests remain green (7 passed). |
There was a problem hiding this comment.
0 issues found across 1 file (changes from recent commits).
Shadow auto-approve: would auto-approve. A focused bug fix: file-like objects now use fallback charset detection instead of stripping undecodable characters, with tests validating the corrected behavior on tricky streams.
Re-trigger cubic
…coding-file-like # Conflicts: # CHANGELOG.md
There was a problem hiding this comment.
0 issues found across 1 file (changes from recent commits).
Shadow auto-approve: would auto-approve. Focused bug fix that makes file-like-object decoding mirror the existing file-path branch, with new tests pinning the corrected fallback detection, truncated-tail, and boundary-split behavior. No rollout, contract, policy, or operational-tradeoff changes.
Re-trigger cubic
The 0.26.1 release landed upstream while this branch was in review; check-version (version-sync.sh) requires the top CHANGELOG entry to be unreleased. Bumps both CHANGELOG.md and unstructured/__version__.py. Signed-off-by: fei <204683769+feiiiiii5@users.noreply.github.com>
There was a problem hiding this comment.
0 issues found across 2 files (changes from recent commits).
Shadow auto-approve: would auto-approve. Targeted bug fix to align file-like decoding with the file-path branch; tests lock in correct behavior for edge cases. No changes beyond version/changelog metadata; this is purely a decoding correction.
Re-trigger cubic
2 participants
Summary
FileTypeDetectionContext.text_head()applied fallback character-set detection only to file paths. For file-like objects it decoded witherrors="ignore", silently stripping undecodable characters — corrupting the text head (and downstream file-type classification) for non-UTF-8 streams such as S3/GCS objects and API uploads.Fix
Mirror the file-path branch: decode strictly with the declared encoding, and on failure fall back to the existing
detect_file_encoding(file=content)(charset-normalizer + common-encoding fallback). Also resolves the TODO that flagged this gap.Test
and_it_uses_character_detection_to_correct_a_wrong_encoding_arg_for_a_file_like_object(previously asserted the silent-empty-string behavior) to expect the corrected text head, matching the file-path case.and_it_detects_a_non_utf8_file_like_object_instead_of_stripping_its_characters(latin-1BytesIOregression for bug/fallback-encoding-detection-for-file-like-objects #4434).Verify
uv run --no-sync pytest 'test_unstructured/file_utils/test_filetype.py::Describe_FileTypeDetectionContext' -qLocal results: the two new/updated tests pass;
ruff check/ruff format --checkclean on changed files. Note: ~24mimetype_magic_detection...tests fail locally because libmagic is not installed on this macOS box — they fail identically onmain(pre-existing environmental failures, unrelated to this change).__version__.pybumped to 0.26.1 and a CHANGELOG entry added per the contribution checklist.AI Disclosure
Root-cause analysis and the initial patch were AI-assisted; the fallback approach was verified against the existing
detect_file_encoding()helper (which already accepted afile=argument), the test was corrected to latin-1-encodable content, and the final diff was human-reviewed.