Skip to content

fix(desktop): paste screenshots and deliver image attachments to the model - #234

Merged
VickyXAI merged 2 commits into
release/desktop-0.2.0-beta.9from
fix/desktop-paste-screenshots
Nov 3, 2026
Merged

VickyXAI merged 2 commits into
release/desktop-0.2.0-beta.9from
fix/desktop-paste-screenshots

Conversation

@KillerQueen-Z

Copy link
Copy Markdown
Collaborator

Problem and resulting behavior

Desktop beta.9 ignores screenshot paste, and even images added through the existing file picker are dropped by agent.send before reaching the agent. Users can now paste a screenshot with Cmd+V / Ctrl+V, preview or remove it, and send it to the model, including an image-only message. Ordinary text paste remains unchanged.

Implementation

  • Share the existing image compression path between file selection and paste. Disable sending while preparation is pending; prevent a removed or older image read from restoring a stale attachment.
  • Validate one inline PNG/JPEG/WebP/GIF up to 4 MB at the serve boundary. Reject filesystem paths, remote URLs, malformed base64 and mismatched image signatures without invoking a model.
  • Carry typed image blocks alongside text through the input queue and agent history; preserve images on /retry, recognize vision input during routing, and use a default prompt for image-only messages.
  • Live testing exposed a related grounding failure: the text-only evaluator did not know the user supplied an image, and forced unnecessary Read calls after a correct visual response. Pass image provenance to that check, retain external/live-data grounding requirements, and stop the tool summary at structured user-message boundaries. The text-only evaluator does not claim to verify visual accuracy.

Validation

  • Reproduced missing native screenshot paste on the public macOS beta.9 app.
  • Local native Electron 44 + this checkout's actual backend + real paid Claude Haiku 4.5 via Solana; no mocked model or socket responses. Wrote a rendered screenshot to the OS clipboard, pasted using Cmd+V, preserved the text draft, removed/re-pasted the image, and submitted through chat. The model read the unpredictable image-only label ORBIT-970862 and correctly identified the blue rectangle and red circle.
  • File-picker/image-only submission also reached the paid model and returned the same label. Plain-text native paste still works; paste does not send automatically.
  • All 937 backend regression tests pass. Complete Desktop test command passes (108 unit tests plus native security, cloud and team-proxy checks); lint and both builds pass.
  • New integration tests cover the real serve queue/agent loop up to a fixture gateway, image-only input, /retry, ordinary follow-up text, and rejection before model invocation. These automated fixture tests are separate from the paid native verification above.
  • Native runtime tested locally on macOS. Windows packaging/checks run in CI; Windows native Ctrl+V has not been locally exercised.

Independent of #233 and based directly on the current Desktop beta.9 release branch. This PR does not publish an installer.

KillerQueen-Z and others added 2 commits November 2, 2026 13:57
stripLargeImageData only handled images nested in tool_result, so a
pasted or picked image (top level of the user message) was written to
the session jsonl in full on every attach. Strip it above the same 50 KB
threshold; the in-memory turn keeps the bytes, and the placeholder tells
the model to ask for a re-attach on resume.
@VickyXAI
VickyXAI merged commit f77a860 into release/desktop-0.2.0-beta.9 Nov 3, 2026
4 checks passed
@VickyXAI
VickyXAI deleted the fix/desktop-paste-screenshots branch November 3, 2026 00:21
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants