Describe the bug
Currently, when trying to determine a file type from its initial text (text_head), the parser attempts fallback character-set detection only for physical file paths. For file-like objects (e.g., SpooledTemporaryFile, BytesIO streams), it blindly decodes the first 4096 bytes using the default encoding (usually UTF-8) with errors="ignore". This can strip characters and lead to corrupted text or incorrect filetype classification when ingesting documents from cloud storage (S3/GCS) or APIs.
To Reproduce
Pass a non-UTF-8 encoded BytesIO object to detect_filetype.
Expected behavior
File-like objects should use the same detect_file_encoding fallback mechanism as physical file paths to avoid data loss on decoding errors.
Screenshots
N/A
Environment Info
N/A - General bug in filetype.py
Additional context
Resolves the TODO in filetype.py:text_head.
Describe the bug
Currently, when trying to determine a file type from its initial text (
text_head), the parser attempts fallback character-set detection only for physical file paths. For file-like objects (e.g.,SpooledTemporaryFile,BytesIOstreams), it blindly decodes the first 4096 bytes using the default encoding (usually UTF-8) witherrors="ignore". This can strip characters and lead to corrupted text or incorrect filetype classification when ingesting documents from cloud storage (S3/GCS) or APIs.To Reproduce
Pass a non-UTF-8 encoded
BytesIOobject todetect_filetype.Expected behavior
File-like objects should use the same
detect_file_encodingfallback mechanism as physical file paths to avoid data loss on decoding errors.Screenshots
N/A
Environment Info
N/A - General bug in
filetype.pyAdditional context
Resolves the TODO in
filetype.py:text_head.