Wrap file operations with fsspec to support multiple storage backends

[titusz]

Summary

Add support for remote/cloud file processing by integrating fsspec as a lightweight file resolution layer. This allows all public API functions to accept fsspec-compatible URIs (e.g., s3://, https://, gcs://) in addition to local file paths, with full backward compatibility.

Example usage after implementation:

import iscc_sdk as idk

# Local paths work exactly as before (zero overhead)
idk.code_iscc("/local/file.mp4")

# Remote URIs are transparently downloaded and processed
idk.code_iscc("s3://my-bucket/file.mp4")
idk.code_iscc("https://example.com/document.pdf")

Research Findings

universal_pathlib (UPath) was evaluated and rejected as a drop-in approach because:

• Remote UPath instances do NOT subclass pathlib.Path and are NOT os.PathLike
• Path(remote_upath) raises TypeError
• open(remote_upath, "rb") raises TypeError
• External tools (FFmpeg, fpcalc, taglib) require local filesystem paths regardless

fsspec alone is the right tool — lightweight (~200KB), zero transitive dependencies, and provides exactly the file resolution we need without requiring changes to internal modules.

Constraints

• All underlying processing tools (FFmpeg, fpcalc, taglib, tika, Pillow) require local filesystem access
• Remote files must always be fully downloaded before processing — streaming is not possible
• The feature must be fully backward compatible — existing str | Path inputs must work identically

Implementation Plan

Step 1: Add fsspec dependency

In pyproject.toml, add fsspec to dependencies:

dependencies = [
    ...
    "fsspec",
]

fsspec has zero transitive dependencies and adds ~200KB.

Step 2: Add local_path context manager to utils.py

Add a context manager that transparently resolves any fsspec-compatible URI to a local file path:

@contextmanager
def local_path(fp):
    """Resolve an fsspec-compatible URI or local path to a local file path.

    For local paths, yields the path directly with zero overhead.
    For remote URIs (s3://, https://, gcs://, etc.), downloads to a
    temporary file, yields the temp path, and cleans up afterward.

    :param fp: Local file path or fsspec-compatible URI string.
    :yields: Path object pointing to a local file.
    """
    import fsspec.utils
    fp_str = str(fp)
    protocol, _ = fsspec.utils.get_protocol(fp_str)
    if protocol in ("file", ""):
        yield Path(fp_str)
    else:
        suffix = Path(fsspec.utils.stringify_path(fp_str).split("/")[-1]).suffix
        with tempfile.NamedTemporaryFile(suffix=suffix, delete=False) as tmp:
            tmp_path = Path(tmp.name)
        try:
            with fsspec.open(fp_str, "rb") as remote:
                with open(tmp_path, "wb") as local:
                    shutil.copyfileobj(remote, local)
            yield tmp_path
        finally:
            tmp_path.unlink(missing_ok=True)

Key design decisions:

• Zero overhead for local paths — just a protocol string check and yield
• Preserves file extension in temp file (needed for media type detection)
• Cleanup is guaranteed via finally block
• Uses shutil.copyfileobj for memory-efficient streaming download
• Import fsspec.utils lazily inside the function to avoid import-time cost for local-only usage

Export local_path from __all__ in utils.py.

Step 3: Wrap public entry points in main.py

Wrap all 13 code_* functions that accept fp parameter. The change is minimal — one additional indentation level at the top of each function:

def code_iscc(fp, name=None, description=None, meta=None, **options):
    with idk.local_path(fp) as fp:
        fp = Path(fp)
        # ... rest of existing code unchanged ...

Functions to wrap (all in main.py):

1. code_iscc
2. code_iscc_mt
3. code_meta
4. code_content
5. code_text
6. code_text_semantic
7. code_image
8. code_image_semantic
9. code_audio
10. code_video
11. code_data
12. code_instance
13. code_sum

Step 4: Wrap public entry points in metadata.py

Wrap the 2 metadata functions:

1. extract_metadata — wrap fp with local_path
2. embed_metadata — wrap fp with local_path (note: outpath remains local-only, which is correct since we're writing output)

Step 5: Update type annotations

Update docstrings and type comments for all wrapped functions to indicate URI support:

def code_iscc(fp, ...):
    # type: (str | Path, ...) -> idk.IsccMeta
    """
    ...
    :param fp: Local file path, Path object, or fsspec-compatible URI (s3://, https://, etc.)
    ...
    """

Step 6: Add tests

Add test_fsspec.py with tests:

• test_local_path_with_path_object — verify local Path passthrough (no temp file created)
• test_local_path_with_string — verify local string passthrough
• test_local_path_with_memory_uri — use fsspec.filesystem("memory") to test remote path resolution without needing real cloud credentials
• test_local_path_cleanup — verify temp file is removed after context exit
• test_local_path_cleanup_on_error — verify cleanup happens even on exceptions
• test_code_iscc_with_memory_uri — integration test using memory filesystem

Use fsspec's built-in memory:// filesystem for all tests (no network, no credentials, no mocking).

Step 7: Update documentation

• Add fsspec URI examples to the main API docstrings
• Mention remote file support in changelog

Scope Boundaries

In scope:

• local_path context manager in utils.py
• Wrapping main.py (13 functions) and metadata.py (2 functions)
• Tests using memory:// filesystem
• Documentation updates

Out of scope (future work):

• Progress callbacks for large downloads
• Caching of downloaded remote files
• Wrapping lower-level functions (text.py, image.py, etc.) — they remain local-only internal APIs
• CLI support for remote URIs (can be added later by wrapping CLI commands)
• outpath parameter support for remote destinations in embed_metadata