design(flows): structure tree drafted from truncated page text (1200 chars), diverging from PageIndex full-page construction

[keli-wen]

Summary

QuantMind's paper structure tree (the "PageIndex-style" PaperStructureTree produced by PaperFlow(PaperStructureCfg).build) is drafted by a model that only ever sees, per page, the first page_text_chars characters (default 1200) plus the deterministically-extracted headings. The reference implementation this feature is modeled on — VectifyAI/PageIndex — never truncates page bodies: it feeds the model each page's full text (batched into token-bounded windows) and verifies titles against real page content. This issue records the divergence and the resulting shortcomings as a future development point. It is not a blocker for current functionality (node content itself is filled from full page text), and it is unrelated to the usage-tracking work that surfaced it.

How QuantMind builds the tree today

Three stages (see contexts/design/mind/retrieval.md):

1. Deterministic outline signals — extract_outline_signals (quantmind/preprocess/outline.py) scans the whole document and emits all heading candidates + table-of-contents pages + printed-to-physical page offset. Full, not truncated.
2. Model draft structuring — quantmind/flows/paper/_structure.py sends the model {outline: <all headings>, pages: [{page_number, text: page.text[:page_text_chars]}]} and gets back the tree skeleton (title + inclusive page span + one-line summary per node, plus a quality rating). This is the only stage that truncates, at page.text[: cfg.page_text_chars], with page_text_chars defaulting to 1200 (quantmind/configs/structure.py, alongside max_depth=6, max_nodes=128).
3. Deterministic content fill — PaperStructureTree.from_draft (quantmind/knowledge/paper.py) populates each leaf node's content by joining the full text of its cited pages. Full text.

So the model decides which pages belong to which section, and writes each node's summary, from headings + the first ~1200 chars (~200-300 words, often under half a dense page) of each page — in a single truncated pass.

How official PageIndex builds the tree

From the source (pageindex/page_index.py) and docs:

• Full page text, windowed by page count, never truncated. page_list_to_group_text(..., max_tokens=20000, overlap_page=1) concatenates each page's complete text into ~20k-token windows with a 1-page overlap; there is no per-page character/token slicing anywhere in the source.
• TOC-first, generate as fallback. find_toc_pages detects a table of contents in the first ~20 pages; process_toc_with_page_numbers maps printed page numbers to physical indices, process_toc_no_page_numbers fuzzy-matches titles to pages; when there is no TOC (the usual case for arXiv papers), process_no_toc → generate_toc_init / generate_toc_continue infer the structure from full page-text windows.
• Depth from the document's own nesting plus recursive splitting of oversized nodes (max-pages-per-node ≈ 10, max-tokens-per-node ≈ 20000), not a fixed max depth.
• Title verification against real content. check_title_appearance confirms each extracted title actually appears on its assigned page; low accuracy triggers retry / a more conservative fallback.
• Summaries in a dedicated pass over full section text. add_node_text attaches each node's section text from its page range, then generate_summaries_for_structure summarizes from that full text — not from a truncated draft view.

The divergence, and why it is a quality risk

• Structure fidelity. Deciding page spans from headings + the first 1200 chars breaks when a section starts in the lower part of a dense page, when the deterministic heading extractor misses a heading, or when structure must be inferred from body prose. A wrong page span then feeds the wrong pages into stage 3's content fill.
• Summary fidelity. Node summaries are written from the same truncated view, so they can describe only the first portion of a section; PageIndex writes summaries from full section text in a separate pass.
• It only looks fine on sparse inputs. The golden test fixture has ~800 chars/page, so truncation never triggers there — the gap is invisible on that fixture and appears on real, dense papers. (A structure build on the fixture reports ~1.1-1.5k input tokens precisely because the whole prompt is instructions + outline + four short pages.)

Content is not truncated (stage 3 uses full page text), so this is a structure / summary fidelity gap, not a content-truncation bug.

Possible future directions

• Feed full page text in token-bounded windows (PageIndex's approach): bound the prompt by pages-per-window (e.g. ~20k tokens, 1-page overlap), not by clipping each page. Cap cost by page count, not by character truncation.
• Add a title-verification pass (check_title_appearance-style) and, for documents that have one, TOC detection with printed→physical page mapping.
• Generate node summaries in a dedicated pass over full section text (stage 3 already reads it) rather than during the truncated draft.
• At minimum, raise the page_text_chars default so a typical page is not clipped, and document when truncation is safe (sparse pages / clean TOC).

References

• QuantMind: quantmind/flows/paper/_structure.py, quantmind/configs/structure.py, quantmind/preprocess/outline.py, quantmind/knowledge/paper.py (from_draft), contexts/design/mind/retrieval.md.
• PageIndex: https://github.com/VectifyAI/PageIndex, source pageindex/page_index.py, and the document-processing flow.