fix(security): bound quadratic array-stream decoding in is_pdf_too_complex (SEC-146) - #4437
Conversation
…mplex (SEC-146) is_pdf_too_complex re-implemented pypdf's array-based /Contents decoding using the same O(n^2) `raw_data += obj.get_data()` accumulation pattern patched upstream under CVE-2026-33123 (GHSA-qpxp-75px-xjcp). A crafted PDF with many small stream objects in a page's /Contents array could force excessive CPU/memory on this untrusted-input path. - Accumulate into a bytearray with .extend() instead of rebinding bytes. - Cap total accumulated bytes per page at MAX_RAW_STREAM_BYTES (50 MB) as defense-in-depth, mirroring pypdf's own fix. - Bump the pinned pypdf dependency to >=6.9.1 so the library's own code path is patched too. - Add regression tests: bounded-time accumulation over 15k small streams, and early-break enforcement of the byte cap. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
…C-146) Addresses findings from three peer reviews of the initial fix: - Dereference an indirect /Contents up front. DictionaryObject.get (unlike __getitem__) does not resolve references, so a page whose /Contents was an indirect array of streams silently skipped the array branch -- both the complexity heuristic and the new caps were bypassed. (reviewer C, confirmed) - Add an array-entry cap (max_content_stream_array_entries=10_000, pypdf's number). A byte cap alone does not bound an array of many empty/tiny streams; each entry still forces an object resolve + get_data. (reviewers A, C) - Add a document-level decoded-byte budget (max_total_stream_bytes=200MB). The per-page cap left total work at pages x per-page cap, and pages can share one indirect /Contents array, so a ~30KB file could force minutes of scanning. (reviewer B, High, confirmed: 31KB/8-page file -> 3.5s pre-budget) - Check the byte budget before extending, so an oversized single stream is never fully materialized into the accumulator and scanned. (reviewer A) - Fail closed: exceeding any resource limit now returns True (too complex -> skip PDFMiner) instead of classifying a truncated prefix. - Count operators with finditer instead of findall, so counting no longer allocates a match list proportional to stream size. (reviewers B, C) - Promote the caps to keyword parameters with module-constant defaults, matching the existing min_* knobs, and document them. Tests rewritten around real PdfWriter/PdfReader fixtures (FlateDecode streams so the file stays small while decoded output is huge -- the real attack shape): direct + indirect graphics-heavy arrays, a many-small-streams array that fails on the pre-fix code, the entry cap, the pre-copy byte cap, and the cross-page budget. All deterministic; the one timing assertion has ~200x headroom. Note: reviewer B flagged a separate O(n^2) string-accumulation pattern in unstructured/partition/html/transformations.py (element merge loop) -- out of SEC-146's scope (pdf.py / pdf_image/), to be tracked separately. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Addresses the follow-up reviews of af7617d. reviewer A (Medium, confirmed): the document byte budget was charged once per fully-decoded page, after the array loop, so a page whose array was [valid stream, stream that raises] hit `except: continue` and discarded the accounting -- a ~2 KB file with many such pages restored unbounded cross-page work (10 pages -> 900 KB decoded, budget never fired). The budget is now charged incrementally as each chunk is decoded, before the copy and before a sibling can raise, so partial-page work always counts. reviewer B (Medium A): the document budget failed closed, which for an aggregate over ordinary pages can silently strip the text layer from a large legitimate PDF (~419 pages at 500 KB/page). Per the author decision, kept fail-closed but raised the default well above any plausible real document (200 MB -> 1 GB; heaviest real page is ~31 MB) and log at warning so the rare event is visible. reviewer B (Medium B): the byte budget did not bound an array of many zero-byte entries -- each costs a get_data() but adds no bytes, so page count multiplied work again (2,000 pages -> 4.68 s, budget never moved). Added a document-level entry budget (max_total_array_entries, 1,000,000) charged per entry. Also from the reviews: - Non-array branch no longer copies the stream into a bytearray (aliases the bytes directly; the regexes accept bytes). (reviewer B note) - Docstring summary and the "materialized" wording corrected: an oversized stream is never accumulated or regex-scanned, but get_data() does decode it once (matching pypdf's own order). (reviewers B, C notes) - Test fixture documents why it uses pypdf private API (writing an already-compressed stream to keep the file small). (reviewers B, C notes) New tests: document-budget survives mid-page decode errors, and the entry budget bounds a shared many-empty-entry array across pages. Full test_pdf.py: 178 passed, 1 skipped. Not changed (deliberately): operator counting still counts all matches rather than early-stopping at max_graphics_ops -- early-stop would under-count the graphics:text ratio and change classification; CPU is already bounded by the byte caps. (reviewer C optional note) Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
There was a problem hiding this comment.
All reported issues were addressed across 6 files
Shadow auto-approve: would not auto-approve because issues were found.
Re-trigger cubic
… (SEC-146) Addresses cubic review comments on #4437. - P1 (pdf.py): a stream that pypdf refuses to decode because its output exceeds pypdf's own per-filter limit (LimitReachedError -- e.g. a FlateDecode compression bomb capped at ZLIB_MAX_OUTPUT_LENGTH, 75 MB) was caught by the generic `except Exception: continue` and the page was silently skipped, so the file failed open and the bomb was handed to PDFMiner. Now caught specifically and failed closed (return True), consistent with the other per-stream caps. Other decode errors still skip the page (fail open) so one odd/unsupported stream doesn't flag an otherwise-fine PDF. - The related comment about decoding before the byte cap is resolved by the same change: the declared /Length is the compressed size (tiny for a bomb) so it can't gate the decode earlier, and pypdf already bounds each decode at its 75 MB limit and raises -- which now fails closed rather than spiking memory. - P3 (test): removed dead `BigStream.calls` attribute/increment that no assertion read. New test (parametrized over array and standalone /Contents): a stream raising LimitReachedError makes is_pdf_too_complex return True. Full test_pdf.py: 180 passed, 1 skipped. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
There was a problem hiding this comment.
0 issues found across 2 files (changes from recent commits).
Shadow auto-approve: would require human review. Introduces fail-closed caps that flag large content streams as too complex, skipping text extraction; a human should validate these thresholds and the product impact on legitimate large PDFs.
Re-trigger cubic
… (SEC-146) Review finding: the document entry budget (max_total_array_entries) was charged only after an array item resolved to an object with get_data(), so non-stream entries (NullObject, dicts) were resolved and traversed but never counted. A crafted PDF sharing one array of up to 10,000 non-stream objects across many pages kept each page under the per-page cap while the document budget never advanced -- CPU work scaled with page count again (reproduced: 1,000 pages x 9,999 nulls, ~10M traversals, returned False). Charge len(contents) against the entry budget up front, before resolving or decoding any entry, and check it before the loop. This counts every slot (stream or not) and avoids iterating an array that already exceeds the budget. The byte budget is still charged incrementally per decoded stream, so the mid-page decode-error accounting is unchanged. New regression test: a shared array of NullObjects across 200 pages with max_total_array_entries=1 now fails closed (was False on the pre-fix code). Full test_pdf.py: 181 passed, 1 skipped. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
There was a problem hiding this comment.
0 issues found across 3 files (changes from recent commits).
Shadow auto-approve: would require human review. Adds new fail-closed caps and threshold behavior for content streams; choosing these limits is an operational tradeoff a human should review for impact on legitimate PDFs.
Re-trigger cubic
… (SEC-146) Comment/docstring/CHANGELOG wording only; no behavior change. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
There was a problem hiding this comment.
0 issues found across 3 files (changes from recent commits).
Shadow auto-approve: would require human review. Adds new per-page/document caps on decoded bytes and array entries to the complexity heuristic, changing behavior for large legitimate PDFs; a human should validate the thresholds.
Re-trigger cubic
Cut this as the 0.26.1 release rather than an unreleased dev cycle, per review. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
There was a problem hiding this comment.
0 issues found across 2 files (changes from recent commits).
Shadow auto-approve: would require human review. Introduces resource caps and fail-closed behavior on a core PDF path; flagging legitimate large PDFs as too complex is a product/operational tradeoff. Full implementation not fully visible due to truncated diff.
Re-trigger cubic
|
a few comments:
|
… (SEC-146) Addresses PR review (badGarnet): - High: small compressed files bypassed every cap via the min_file_size_bytes gate (a 55 KB file can declare ~900 MB of content). Default is now 0 (inspect every file); skipping small files is opt-in. Inspection is bounded, and normal small files stay cheap (~0.4 ms) via the min_raw_stream_bytes per-page skip. - Medium: a decode error on one array stream skipped the whole page (fail open), so later streams evaded the byte limits. Decode is now per-item: a bomb still fails closed (LimitReachedError -> True), but an unreadable stream skips only itself and the rest of the page is inspected and charged. - Medium (peak memory during decode): not separately fixable here. get_data() must decode to know a stream's size, and a single stream's decode is already bounded by pypdf's own 75 MB per-filter limit, which raises LimitReachedError and now fails closed. pypdf exposes no per-call output limit and lowering its module-global limit is not thread-safe. New tests: a small (<1 MB) array-bomb flagged by default, and a bad array stream that no longer masks an over-cap sibling. Full test_pdf.py: 183 passed, 1 skipped. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
There was a problem hiding this comment.
0 issues found across 3 files (changes from recent commits).
Shadow auto-approve: would require human review. Changes the default min_file_size_bytes to 0, expanding the complexity check to all files, and introduces new resource caps and a pypdf version bump. The performance/security tradeoff of inspecting every PDF deserves human sign-off.
Re-trigger cubic
|
| CVE | Package | Version | Fix |
|---|---|---|---|
| CVE-2026-45829 | chromadb |
1.5.7 |
— |
| CVE-2026-54058 | pillow |
12.2.0 |
12.3.0 |
🔶 High · 38 findings
| CVE | Package | Version | Fix |
|---|---|---|---|
| CVE-2026-54280 | aiohttp |
3.13.5 |
3.14.1 |
| CVE-2026-54059 | pillow |
12.2.0 |
12.3.0 |
| CVE-2026-54275 | aiohttp |
3.13.5 |
3.14.1 |
| CVE-2026-59935 | pypdf |
6.10.0 |
6.14.2 |
| CVE-2026-54278 | aiohttp |
3.13.5 |
3.14.1 |
| CVE-2026-59199 | pillow |
12.2.0 |
12.3.0 |
| CVE-2026-59939 | httplib2 |
0.31.2 |
0.32.0 |
| CVE-2026-59884 | pyasn1 |
0.6.3 |
0.6.4 |
| CVE-2026-47265 | aiohttp |
3.13.5 |
3.14.0 |
| CVE-2026-59198 | pillow |
12.2.0 |
12.3.0 |
| CVE-2026-59936 | pypdf |
6.10.0 |
6.14.1 |
| CVE-2026-45134 | langsmith |
0.7.29 |
0.8.0 |
| CVE-2026-34993 | aiohttp |
3.13.5 |
3.14.0 |
| CVE-2026-55380 | pillow |
12.2.0 |
12.3.0 |
| CVE-2026-59200 | pillow |
12.2.0 |
12.3.0 |
| CVE-2026-69244 | aiohttp |
3.13.5 |
3.14.3 |
| CVE-2026-54060 | pillow |
12.2.0 |
12.3.0 |
| CVE-2026-54277 | aiohttp |
3.13.5 |
3.14.1 |
| CVE-2026-59885 | pyasn1 |
0.6.3 |
0.6.4 |
| CVE-2026-41066 | lxml |
6.0.3 |
6.1.0 |
| CVE-2026-59886 | pyasn1 |
0.6.3 |
0.6.4 |
| CVE-2026-50269 | aiohttp |
3.13.5 |
3.14.0 |
| CVE-2026-59203 | pillow |
12.2.0 |
12.3.0 |
| CVE-2026-59937 | pypdf |
6.10.0 |
6.14.0 |
| CVE-2026-49477 | soupsieve |
2.8.3 |
2.8.4 |
| CVE-2026-54279 | aiohttp |
3.13.5 |
3.14.1 |
| CVE-2026-69247 | cryptography |
46.0.7 |
50.0.0 |
| CVE-2026-59197 | pillow |
12.2.0 |
12.3.0 |
| CVE-2026-49476 | soupsieve |
2.8.3 |
2.8.4 |
| CVE-2026-54274 | aiohttp |
3.13.5 |
3.14.1 |
| CVE-2026-44843 | langchain-core |
1.2.28 |
1.3.3 |
| CVE-2026-59205 | pillow |
12.2.0 |
12.3.0 |
| CVE-2026-44432 | urllib3 |
2.6.3 |
2.7.0 |
| CVE-2026-54273 | aiohttp |
3.13.5 |
3.14.1 |
| CVE-2026-59204 | pillow |
12.2.0 |
12.3.0 |
| CVE-2026-48526 | pyjwt |
2.12.1 |
2.13.0 |
| CVE-2026-55379 | pillow |
12.2.0 |
12.3.0 |
| CVE-2026-69249 | cryptography |
46.0.7 |
49.0.0 |
🟡 Medium · 33 findings
| CVE | Package | Version | Fix |
|---|---|---|---|
| CVE-2026-44512 | onnx |
1.21.0 |
1.22.0 |
| CVE-2026-54531 | pypdf |
6.10.0 |
6.13.0 |
| CVE-2026-71870 | pypdf |
6.10.0 |
6.15.0 |
| CVE-2026-41425 | authlib |
1.6.9 |
1.6.11 |
| CVE-2025-3000 | torch |
2.10.0 |
2.13.0 |
| CVE-2026-54276 | aiohttp |
3.13.5 |
3.14.1 |
| CVE-2026-69243 | aiohttp |
3.13.5 |
3.14.2 |
| CVE-2026-41182 | langsmith |
0.7.29 |
0.7.31 |
| CVE-2026-71852 | pypdf |
6.10.0 |
6.15.0 |
| CVE-2026-55798 | pillow |
12.2.0 |
12.3.0 |
| CVE-2026-48735 | pypdf |
6.10.0 |
6.12.1 |
| CVE-2026-47751 | anthropics/claude-code-action |
beta |
1.0.74 |
| CVE-2026-59890 | setuptools |
82.0.1 |
83.0.0 |
| CVE-2026-71554 | h2 |
4.3.0 |
4.4.1 |
| CVE-2026-48523 | pyjwt |
2.12.1 |
2.13.0 |
| CVE-2026-48525 | pyjwt |
2.12.1 |
2.13.0 |
| CVE-2026-41168 | pypdf |
6.10.0 |
6.10.1 |
| CVE-2026-41314 | pypdf |
6.10.0 |
6.10.2 |
| CVE-2026-41479 | authlib |
1.6.9 |
1.6.10 |
| CVE-2026-41312 | pypdf |
6.10.0 |
6.10.2 |
| CVE-2026-45409 | idna |
3.11 |
3.15 |
| CVE-2026-69248 | cryptography |
46.0.7 |
49.0.0 |
| CVE-2026-49461 | pypdf |
6.10.0 |
6.12.2 |
| CVE-2026-44681 | authlib |
1.6.9 |
1.6.12 |
| CVE-2026-59881 | aiohttp |
3.13.5 |
3.14.2 |
| CVE-2026-44431 | urllib3 |
2.6.3 |
2.7.0 |
| CVE-2026-54651 | pypdf |
6.10.0 |
6.13.1 |
| CVE-2026-41313 | pypdf |
6.10.0 |
6.10.2 |
| CVE-2026-41481 | langchain-text-splitters |
1.1.1 |
1.1.2 |
| CVE-2026-48155 | pypdf |
6.10.0 |
6.12.0 |
| CVE-2026-59938 | pypdf |
6.10.0 |
6.14.0 |
| CVE-2026-54530 | pypdf |
6.10.0 |
6.13.0 |
| CVE-2026-48522 | pyjwt |
2.12.1 |
2.13.0 |
🟢 Low · 4 findings
| CVE | Package | Version | Fix |
|---|---|---|---|
| CVE-2026-48156 | pypdf |
6.10.0 |
6.12.0 |
| CVE-2026-44405 | paramiko |
4.0.0 |
— |
| CVE-2026-48524 | pyjwt |
2.12.1 |
2.13.0 |
| CVE-2026-49460 | pypdf |
6.10.0 |
6.12.2 |
View full analysis in Upwind Console
Scan completed in 1m 17s
Scan history (2 scans)
| Commit | Scanned at | New | Resolved | Net |
|---|---|---|---|---|
4dd9d50 |
2026-08-14 21:26 UTC | +77 | 0 | +77 |
c8b495b < |
2026-08-14 21:35 UTC | +77 | 0 | +77 |
Last scanned: c8b495b · 2026-08-14 21:35 UTC
|
| Rule | Resource | File |
|---|---|---|
| 'apk add' is missing '--no-cache' | — | Dockerfile |
🟡 Medium · 1 finding
| Rule | Resource | File |
|---|---|---|
| ':latest' tag used | — | Dockerfile |
🟢 Low · 1 finding
| Rule | Resource | File |
|---|---|---|
| No HEALTHCHECK defined | — | Dockerfile |
View full analysis in Upwind Console →
Scan completed in 45s
Scan history (2 scans)
| Commit | Scanned at | New | Resolved | Net |
|---|---|---|---|---|
4dd9d50 |
2026-08-14 21:26 UTC | +3 | 0 | +3 |
c8b495b < |
2026-08-14 21:36 UTC | +3 | 0 | +3 |
Last scanned: c8b495b · 2026-08-14 21:36 UTC
|
Thanks @badGarnet — all three reproduced and addressed in c8b495b. 1. Small files bypass the new limits (High). Fixed. Tradeoff worth a look: this reverts the small-file fast path from #4268 (~0.44 ms/PDF on the partition path). The knob remains, so a perf-sensitive caller can set 2. One decode error skips the rest of the page (Medium). Fixed. Decoding is now per stream: a bomb still fails closed ( 3. Byte cap checked only after decode (Medium). Confirmed, but bounded and not separately fixable here. Full |
Summary
Fixes SEC-146.
is_pdf_too_complex()inunstructured/partition/pdf.pyre-implemented pypdf's array-based/Contentsdecoding using the same quadraticraw_data += obj.get_data()accumulation that pypdf patched under CVE-2026-33123 / GHSA-qpxp-75px-xjcp ("Inefficient decoding of array-based streams"). A crafted PDF whose page/Contentsis an array of many small stream objects could force excessive CPU/memory. This function runs on every partitioned PDF, so it sits on the untrusted-input path (pdf.py:309).Per the ticket, this keeps the intentional lightweight raw-bytes approach (added in #4268 to cheaply detect vector-heavy CAD/engineering PDFs) — it does not switch to pypdf's expensive
ContentStream. It just makes the accumulation efficient and bounded.Changes
bytearraywith.extend()instead of rebinding abytesobject (amortized O(1) vs O(n²)).max_raw_stream_bytes(50 MB) — decoded bytes per page, checked before each stream is copied so an oversized stream is never accumulated or regex-scanned.max_content_stream_array_entries(10,000, pypdf's number) — bounds an array of many empty/tiny streams that a byte cap alone misses./Contentsarray → tiny file, unbounded scan). Charged incrementally per stream so a mid-page decode error can't discard the accounting. Set far above any plausible real document and logged atwarning:max_total_stream_bytes(1 GB) — decoded bytes per document.max_total_array_entries(1,000,000) — decoded entries per document (zero-byte entries never advance the byte total but still cost a decode)./Contentsbefore the array check —DictionaryObject.get(unlike__getitem__) does not resolve references, so an indirect array of streams was silently skipping the array branch entirely (both the heuristic and the caps). 87 of 1,105 corpus pages reach their content array this way.finditerinstead offindall, so counting no longer allocates a match list proportional to stream size.pypdfto>=6.9.1(from>=6.6.2) so the library's own code path is patched too; lock resolves 6.10.0.Audit (AC4)
Grepped
unstructured/partition/pdf.pyandpdf_image/for other+=-on-stream-bytes loops — none found; this was the only instance. A separate O(n²) string-accumulation pattern inunstructured/partition/html/transformations.py(element-merge loop) is out of SEC-146's scope (not on the PDF path) and is tracked in its own ticket.Testing
Rewrote the regression tests around real
PdfWriter/PdfReaderfixtures (FlateDecode-compressed streams so the file stays small while decoded output is huge — the actual attack shape). Coverage: direct + indirect graphics-heavy arrays, a many-small-streams array (~2 MB file → ~900 MB decoded) that runs for minutes / OOMs on the pre-fix code and returns in ~0.05 s here, the entry cap, the pre-copy byte cap, the cross-page byte and entry budgets, and budget survival across a mid-page decode error. The key tests were confirmed to fail on the pre-fix code.test_unstructured/partition/pdf_image/test_pdf.py: 178 passed, 1 skippedruff check+ruff format --check: cleanAcceptance criteria
is_pdf_too_complexaccumulates viabytearray, notbytes +=+=-on-stream-bytes copies; findings fixed or ticketedpypdfdependency confirmed ≥ 6.9.1🤖 Generated with Claude Code