Skip to content

bug/fallback-encoding-detection-for-file-like-objects #4434

Description

@ujjwalredd

Describe the bug
Currently, when trying to determine a file type from its initial text (text_head), the parser attempts fallback character-set detection only for physical file paths. For file-like objects (e.g., SpooledTemporaryFile, BytesIO streams), it blindly decodes the first 4096 bytes using the default encoding (usually UTF-8) with errors="ignore". This can strip characters and lead to corrupted text or incorrect filetype classification when ingesting documents from cloud storage (S3/GCS) or APIs.

To Reproduce
Pass a non-UTF-8 encoded BytesIO object to detect_filetype.

Expected behavior
File-like objects should use the same detect_file_encoding fallback mechanism as physical file paths to avoid data loss on decoding errors.

Screenshots
N/A

Environment Info
N/A - General bug in filetype.py

Additional context
Resolves the TODO in filetype.py:text_head.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions