Repository navigation
Conversation
Chat attachments sent inline as attachment_uploads, which the embed widget uses, were parsed by the document reader only. The VLM OCR fallback for scanned or image-only documents ran only for temporary attachments uploaded through /sessions/:id/attachments, so the same scanned PDF was readable in the console chat and empty in an embedded page. Apply the agent's VLM OCR settings (VLMModelID, AttachmentImageUnderstanding, AttachmentOCRMaxPages) to inline attachments too, reusing the temporary document gate: text-poor content with page images is OCR'd and the text is appended before line truncation. Refs Tencent#5365, Tencent#4312 Signed-off-by: Jace Xia <fxia0018@student.monash.edu>
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
A scanned PDF uploaded in an embedded chat comes back with no text, while the same file works in the console chat (#5365).
The two entry points parse attachments differently:
POST /sessions/:id/attachments).UploadTemporaryDocumentstores the agent'sVLMModelID,AttachmentImageUnderstandingandAttachmentOCRMaxPages, and the parse worker OCRs text-poor documents with the VLM (temporaryDocumentService.applyImageUnderstanding).attachment_uploads.AttachmentProcessor.processWithDocumentReaderonly runs the document reader, so a PDF without a text layer yields page images and no text.This PR applies the same OCR fallback to inline attachments:
ProcessAttachmenttakes anAttachmentProcessOptionsstruct instead of a bareasrModelID.qa.gofills it from the agent config, using the same conditions asUploadTemporaryDocument(ImageUploadEnabled,VLMModelID,AttachmentImageUnderstanding,AttachmentOCRMaxPages).service.OCRScannedAttachmentreuses the temporary-document gate and VLM cascade. It runs only when the parsed text is under 200 runes and page images are available. It uses the scanned-PDF prompt and never captions. The existingunderstandImagesWithVLMmethod now delegates to a package function, so its behavior is unchanged.ResolveAndStore, which works on the same image refs. The OCR text is appended to the markdown before line truncation, as in the temporary path.Agents without
AttachmentImageUnderstandingbehave as before. Inline attachments are already parsed synchronously, so OCR adds VLM latency to that request. It is bounded by the existing page cap (default 8) and concurrency settings.Type of Change
Related Issue
Fixes #5365
Refs #4312, which also covers embed uploads not being saved as temporary documents. This PR only fixes the missing scanned-document OCR.
Testing
go test -race -count=1 ./internal/handler/session/passes.go test -count=1 ./internal/application/service/passes exceptTestSkillPythonVerifier, which fails the same way on unmodifiedmainin my environment (local Python setup).TestProcessAttachmentScannedPDFUsesVLMOCRandTestProcessAttachmentScannedPDFWithoutOCROptIndriveProcessAttachmentwith a reader that returns a scanned PDF (page image only, no text). With the opt-in, the OCR text appears in the attachment content. Without it, the VLM is not called.TestOCRScannedAttachment*cover the page cap, refs without inline bytes, text-rich content, and a missing VLM model.gofmtandgit diff --checkpass. I didn't run golangci-lint locally. New lines stay within the 120-columnllllimit.Checklist
git diff --check origin/main...HEADpasses