feat: PDF attachment extraction (metadata, bytes, streaming)

[yonikremer]

Summary

Exposes PDF embedded-file attachments end-to-end through the native parser. Before this change docling-parse silently ignored /EmbeddedFiles and /FileAttachment annotations; consumers had no way to discover or extract attached payloads (spreadsheets, source docs, annotations) without a second PDF library. This PR adds a metadata-only first pass and an on-demand, size-gated byte path so attachments can be listed cheaply and fetched safely even for large files.

---

Why this is needed

• Parity with PDF 1.7 / ISO 32000-2: The spec carries embedded files in two places — the document-level name-tree Catalog/Names/EmbeddedFiles and page-level FileAttachment annotations anchored to a Rect. Both are common in enterprise PDFs (invoice ZUGFeRD/Factur-X, portfolio PDFs, review markup).
• No existing surface in docling-parse: get_annotations/get_page never walked those structures; callers fell back to pypdf/qpdf out-of-band, losing the thread-safe decode cache and timing infrastructure.
• Downstream demand: docling and docling-core need attachment metadata during conversion (preserve or route attachments), and the streaming guard is required to avoid OOM on filter-bombs in the full 110+ doc corpus.
• Safety: Blindly decoding every embedded stream at load time would replicate the corpus into memory. The new design keeps load-time cost at JSON-metadata only.

---

Core design

1. C++ extraction — src/parse/qpdf/attachments.h (new, MIT/SPDX)

• Entry point extract_attachment_records(QPDF&, QPDFObjectHandle root) walks:
  • Root/Names/EmbeddedFiles name-tree recursively: handles both /Names and /Nums arrays and /Kids sub-trees (/Limits implicit via recursion) — the original /Names-only walk missed producer variants.
  • Every page Annots[] where Subtype == /FileAttachment, resolving FS → EF stream.
• Identity & dedup: Streams are keyed by QPDFObjGen (object + generation). A single embedded file referenced from the name-tree and two annotations (or multiple pages) collapses to one AttachmentRecord with multiple AttachmentAnnotations. Direct (non-indirect) EF streams — inline dicts with sentinel (0,0) — are never deduped (each is a distinct inline object) and are flagged !isIndirect() so the byte path can fail loudly rather than chasing a free-list object 0.
• Metadata only at extraction: name from /UF→/F (UTF-8 sanitized), mime_type from stream Subtype, size from Params/Size → DL → Length priority. getStreamDict() helper hides the qpdf quirk on this version where stream handles do not proxy getKeys()/hasKey() — all dict lookups go through getDict().
• JSON bridge: attachments_to_json() emits {name, mime_type, size, annotations: [{page_no, bbox:[x0,y0,x1,y1]}]}; internals like obj_gen never leak to Python (scope-creep fix in round 2).

2. Decoder — src/parse/pdf_decoders/document.h

• Lazy, cached: ensure_attachments_loaded() runs once per pdf_decoder<DOCUMENT> lifetime, stores vector<AttachmentRecord>, reset on process_document_from_bytesio.
• Shared helpers (dedupe refactor): require_attachment_record(index, max_size), require_attachment_stream(index, max_size), warn_if_raw_length_exceeds(stream, max_size) factor the bounds/size/isIndirect/getObjectByObjGen/isStream prelude and the Length filter-bomb warning (10× slack — compression can legitimately expand) that was duplicated between the two byte paths.
• Two byte paths, same policy:
  • get_attachment_data(index, max_size) → Buffer — stream.getStreamData(qpdf_dl_all) then post-decode size check vs max_size.
  • write_attachment_data(index, max_size, path) — stream.pipeStreamData(&LimitedFilePipeline) with an incremental written > max_size throw inside Pipeline::write() so a 9 MB zip with lying /Length OOMs before it spills; finish() flushes. Both paths pre-check rec.size > max_size and the raw /Length heuristic before any decode.
• Error contract: Missing key → RuntimeError("key not found") (unified across get_attachments/get_attachment_data), OOB index → out_of_range, oversize/direct-stream/missing-stream → RuntimeError with human message. No silent swallowing — failures are LOG_S(WARNING) + throw.

3. Python — docling_parse/pdf_parser.py + src/pybind/*

• Models: PdfAttachment(name, mime_type?, size, annotations: List[FileAttachmentAnnotation]), FileAttachmentAnnotation(page_no: int, bbox: BoundingRectangle) — 0-based pages, bottom-left origin, BoundingRectangle polygons match page geometry.
• API:
  • PdfDocument.get_attachments() -> List[PdfAttachment] — does not decode bytes.
  • PdfDocument.get_attachment_data(index, *, max_size: int) -> bytes — max_size required (TypeError if omitted) to force caller policy.
  • PdfDocument.get_attachment_stream(index, *, max_size: int) -> BinaryIO — metadata-driven: if effective_size ≤ 8 MB returns BytesIO; otherwise mkstemp + native write_attachment_data (no 2× memory), wrapped in module-level _AttachmentDeletingFile (munlinks on close(), supports with). size==0 (unknown) falls back to max_size; under-estimates spill after a memory fetch.
• Pybind: docling_parser::get_attachments/get_attachment_data/write_attachment_data keyed by internal doc key, same throw-on-missing-key contract.

4. Key trade-offs

• Decode-on-demand vs eager: Eager would simplify callers but double memory for the corpus; demand keeps processMemoryFile fast.
• Pipe vs buffer: Buffer path remains for small files (fast, no fd); pipe avoids the C++ Buffer allocation for large payloads — measured +9 MB spill no longer holds 2×.
• Dedup by ObjGen not hash: Content hash would collapse different files with same bytes; ObjGen preserves spec identity.

---

Compatibility with docling-core

This PR is complementary to, not dependent on, the unmrged docling-project/docling-core#713

• No hard import: PdfAttachment/FileAttachmentAnnotation (docling_parse/pdf_parser.py:FileAttachmentAnnotation) deliberately stays parser-level and only re-uses the stable core type BoundingRectangle. AttachmentItem is not imported here so main/CI stay green while your branch is unreleased.
• Field alignment is intentional: name/mime_type/size match AttachmentItem 1:1, so docling (the converter) can later do AttachmentItem(name=a.name, mime_type=a.mime_type, size=a.size, target=..., status=...) with zero translation. Parser-only annotations{page_no, bbox} stays on the parse side (core models target/status instead).
• After your core PR merges: docling-parse will bump to docling-core>=2.92,<3 and can ship an optional helper PdfAttachment.to_attachment_item() -> AttachmentItem (no API break). The actual DoclingDocument.attachments population belongs in docling, not here.

A maintainer can merge this PR without waiting for the core branch; the integration point is docling.

Testing

• Synthetic PDFs (tests/test_attachments.py, 5 tests): _build_pdf crafts minimal xref PDFs covering no-annot, one-file-two-annots (ID dedup → 2 annotations on pages 0/2 with exact Rect→BoundingRectangle + polygon asserts), same name/different bytes (must stay 2 records, payload differs), large size-gate (max_size violation) and >8 MB spill (returns file not BytesIO, auto-unlink).
• Full suite: 153 passed in 19:50 on Windows/MinGW GCC 16.1.0 — no regressions in colorspace, shading, transparency, threaded parse/render, locale, or reference-document ground-truth.

---

Files changed (this PR only)

src/parse/qpdf/attachments.h (new), src/parse/pdf_decoders/document.h, src/pybind/docling_parser.h, app/pybind_parse.cpp, docling_parse/pdf_parser.py, tests/test_attachments.py

Excluded from this PR: 27c9397 (threaded-parse Windows key fix) and 8c9eaa1 (CLAUDE.md build notes) — separate PRs if needed.

Co-authored-by: Claude noreply@anthropic.com
Signed-off-by: yoni kremer yoni.kremer@gmail.com

[github-actions]

✅ DCO Check Passed

Thanks @yonikremer, all your commits are properly signed off. 🎉

[mergify]

Merge Protections

🟢 Merge protection satisfied — ready to merge.

Show 1 satisfied protection

🟢 Enforce conventional commit

Make sure that we follow https://www.conventionalcommits.org/en/v1.0.0/

• title ~= ^(fix|feat|docs|style|refactor|perf|test|build|ci|chore|revert)(?:\(.+\))?(!)?:

[yonikremer]

@PeterStaar-IBM Hey, can you review the PR? I think it would be a great feature for docling.

[PeterStaar-IBM]

I think the convention is that in C++ we use 0-based (since vectors are zero based and all memory is stored there), however the python interface should be 1-based (following the PDF convention). The goal of the python is to translate the 1 based to the 0 based.

[PeterStaar-IBM]

no need for this, all these lines need to be removed...

[PeterStaar-IBM]

@yonikremer Thanks for the addition, can you:

1. run the styling: uv run pre-commit run --all-files
2. I would like to see if we need to migrate up the PdfAttachment to docling-core (we dont want to double define it)