Repository navigation
fix(agent-runtime): keep image and blob bytes out of tool result fallback text - #1570
Open
xpeng5278-web wants to merge 1 commit into
Open
xpeng5278-web wants to merge 1 commit into
xpeng5278-web wants to merge 1 commit into
Conversation
…back text An image-only or blob-only host result has no text block, so the MCP content and bare content-block fallbacks stringified the raw payload. That copied base64 into the model text channel, duplicating the image block for vision models and dumping bytes as text for every other model. Strip image data and resource blob bytes in that JSON fallback only. Type, mimeType, and uri stay so the attachment is still described.
muzimu217
reviewed
Oct 11, 2026
muzimu217
left a comment
Contributor
There was a problem hiding this comment.
Verified locally on macOS (patch applied to a detached worktree at current HEAD eec988c0f): agent-runtime full suite 96 files / 1333 tests pass (the +4 over main are the new binary-stripping cases).
The fix is the right companion to #1551, and the helper is cleanly scoped:
withoutInlineBinaryrecurses the fallback JSON, dropping only two things — imagedataand resourceblobbytes — while keepingtype,mimeType,uriand every other field. So an image-only or blob-only tool result no longer floods the context with megabytes of base64, yet the model still sees what the tool returned and where (the resourceurisurvives), which is the actionable part.- Applied on both the bare content-block-array branch and the MCP
contentbranch — the same two paths #1551 touched — so the two PRs stay symmetric. - Image block extraction and the
images/detailssurfaces are untouched, so vision consumers lose nothing.
One boundary note for the record (non-blocking): stripping is on the fallback-text path only — a tool result that returns proper image blocks still feeds them to vision models as blocks. That's the intended split and worth keeping as-is.
Nothing blocking. fix scope, fits the window.
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #1569
What
In the host tool bridge (
packages/agent-runtime/src/runtime.ts), the MCPcontentbranch and the bare content-block-array branch now stringify their no-text fallback through a smallwithoutInlineBinaryhelper. It drops imagedataand resourceblobbytes and keeps everything else (type,mimeType,uri, other fields), so the model is still told what was returned.Image block extraction,
details, theimagesbranch,hostContentBlockTextandtoolResultFromUiare unchanged.Docs:
docs/spec/03-runtime/02-agent-runtime.md§3.3 now states that the no-text JSON fallback also keeps image and blob bytes out of the model-visible text.Why
textPartsonly holdstext,resource.textandresource_linkcontent. An image-only or blob-only result therefore fell back toJSON.stringify(rawContent), base64 included. Vision models received the image twice (image block + base64 text). Text-only models received the base64 as plain text. One screenshot could add hundreds of KB to the context. Theimagesbranch already stripsimagesbefore stringifying, and #1360 / #1551 already keep binary out of the text channel; this aligns the two remaining fallbacks.How tested
packages/agent-runtime/src/runtime.test.ts:contentand image-only bare array → no base64 in text, image block kept,imageCount: 1uriandmimeTypestill presentmain(eec988c0f): the 4 new cases fail (expected '…' not to contain 'cG5nLWJ5dGVz'). With this change:runtime.test.ts311/311 pass.packages/agent-runtime:vitest run→ 1333 tests pass;tsc -p . --noEmitclean.node scripts/check-pr-base-main.mjs,node scripts/check-architecture.mjs,pnpm lint:biome,docs/scripts/check-docs.mjspassed.Scope
Only the no-text JSON fallback in those two branches, plus the regression tests and one spec sentence. Results that have a text block, the
imagesshape, and persisted-row restore are unchanged. Other binary-carrying block kinds (e.g. MCPaudio) are not touched here.