feat(agent): OpenUI bindings — library/fragment builders, toUIOutput, getUiStream() (DEV-773) - #92
Conversation
…r (DEV-773) The SDK half of the OpenUI backport from Noetic (DEV-765). Adds packages/agent/src/lib/openui/: defineComponent/createLibrary (Zod props with normative declaration order), the typed fragment() builder with uiRef/uiState/uiBuiltin, OpenUI Lang expression serialization, and the openui(library) helper producing the wire-shaped plugin preference (Zod -> JSON Schema). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…Stream() (DEV-773) Tools can now author OpenUI render fragments: an optional toUIOutput sibling of toModelOutput on every executable tool shape. Fragments are broadcast as tool.ui_fragment stream events after successful execution (render-only, never sent to the model; a throwing toUIOutput degrades to no-fragment). getUiStream() on ModelResult surfaces UI events across all turns: tool-authored fragments plus the API's response.openui.* wire events (statement/fragment/document) from the openui plugin. Wire events not yet in the SDK's stream-event union arrive via its forward-compat Unknown catch-all, so translation reads the raw payload — the stream works both before and after the SDK regen (DEV-772). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
![devin-ai-integration[bot]](https://avatars.githubusercontent.com/in/811515?s=60&v=4){kind=small}
| export const OPENUI_BUILTIN_COMPONENTS = [ | ||
| 'Action', | ||
| 'Query', | ||
| 'Mutation', | ||
| 'ToolView', | ||
| ] as const; |
There was a problem hiding this comment.
🔍 Declared builtin components are unreachable from the fragment builder
OPENUI_BUILTIN_COMPONENTS (Action, Query, Mutation, ToolView) is documented as implicitly accepted by every library, but fragment() only compiles constructors for components present in library.components (packages/agent/src/lib/openui/fragment.ts:111-113), and createLibrary never seeds the builtins. A tool author therefore cannot construct e.g. ToolView(...) via the typed builder; the only escape hatch, uiBuiltin, prepends @ and would emit @ToolView(...), which is builtin-function syntax rather than a component call. The constant is currently unused anywhere in the package — worth confirming whether the builder is meant to expose these.
Was this helpful? React with 👍 or 👎 to provide feedback.
There was a problem hiding this comment.
Accurate. OPENUI_BUILTIN_COMPONENTS (Action, Query, Mutation, ToolView) is documented as implicitly accepted by every library, but fragment() only compiles constructors for the components a library declares, so ui.Query(...) is not reachable from the typed builder — a caller has to hand-assemble that expression.
Not fixing it here: the builtins are data/action bindings rather than DOM, and the playground renderer returns null for all four, so nothing regresses today. Making them first-class on the builder is a real API addition (they take different argument shapes than a component`s positional props) and belongs with the client event-ingestion work rather than this PR. Leaving open so it is visible rather than silently resolved.
| function unwrapEvent(event: unknown): Record<string, unknown> | null { | ||
| if (!isRecord(event)) { | ||
| return null; | ||
| } | ||
| if (event['isUnknown'] === true && isRecord(event['raw'])) { | ||
| return event['raw']; | ||
| } | ||
| return event; | ||
| } |
There was a problem hiding this comment.
🔍 Unknown-event unwrapping relies on an unverified SDK encoding
unwrapEvent assumes the SDK's forward-compat catch-all materializes as { type: 'UNKNOWN', raw: <original>, isUnknown: true }. @openrouter/sdk is not vendored in this checkout, and no other module in the package inspects isUnknown/raw (the existing forward-compat handling in packages/agent/src/lib/stream-type-guards.ts:139-160 only switches on type). If the generated shape differs (e.g. the raw payload is spread onto the event, or the field is named differently), every response.openui.* wire event would silently translate to null and the UI stream would appear empty pre-regen — exactly the case this code exists to handle. Worth confirming against the installed SDK's Unknown<"type"> runtime encoding.
Was this helpful? React with 👍 or 👎 to provide feedback.
There was a problem hiding this comment.
Fair challenge, and I cannot close it from this checkout either — @openrouter/sdk is a published dependency here, so the { type: "UNKNOWN", raw, isUnknown: true } catch-all shape is an assumption about the SDK`s forward-compat encoding rather than something this PR verifies.
What I can say: unwrapEvent fails safe. If the encoding differs, translateUiEvent returns null and getUiStream() yields nothing for pre-regen wire events — the documented degraded path — rather than throwing or emitting a malformed event. So the blast radius of a wrong guess is "native mode stays silent until the regen lands", which is already its expected state.
Leaving open: the right fix is a test against the real SDK types once the regen adds the OpenUI events, at which point the catch-all branch can be deleted entirely rather than verified.
New private package @openrouter/openui-playground: a local webapp for testing, benching, and evaluating OpenUI generative-UI support. - Progressive renderer over the demo component library (Stack/Card/ Heading/Text/Stat/Badge/Table/Input/Select/Button/Progress) — UI materializes statement-by-statement mid-stream - Two modes with identical event shapes: emulate (local library prompt + reference streaming parser over the text stream — works today) and native (openui() plugin + getUiStream() — flips on when DEV-771/772 land), so the paths can be A/B'd from the history table - Bench stats per run: TTFB, first-statement latency, total time, statement/diagnostic counts, token usage, cost; session history for comparing models and prompts - Reference incremental OpenUI Lang parser (the same logic DEV-770 ports into openrouter-web) with 11 conformance tests - Plain node:http + static client; no build step Verified end-to-end against live models: single-card and 12-statement dashboard prompts parse clean (0 diagnostics) and render progressively. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
| } | ||
| if (def.props !== undefined) { | ||
| component.props = convertZodToJsonSchema(def.props); | ||
| } |
There was a problem hiding this comment.
🔍 Plugin wire schema is generated in output mode, unlike the playground prompt (input mode)
openui() converts prop schemas via convertZodToJsonSchema, which calls z4.toJSONSchema(schema, { target: 'draft-7' }) with the default io: 'output' (packages/agent/src/lib/tool-executor.ts:93-104), while the playground's prompt generator deliberately uses io: 'input' (packages/openui-playground/src/lang/prompt.ts:13-15). For props declared with .default(...), output mode marks the property as required, so the API would see a prop as mandatory that the SDK's own componentProps reports as optional (packages/agent/src/lib/openui/library.ts:88-92). Worth confirming which io mode the API expects before DEV-772 lands, since positional-prop mapping and requiredness are consumed server-side.
Was this helpful? React with 👍 or 👎 to provide feedback.
There was a problem hiding this comment.
Confirmed empirically, and leaving open for the API-side answer you asked for. The divergence is real — z4.toJSONSchema on z4.object({ tone: z4.string().default('info'), label: z4.string() }):
output required: ["tone","label"]
input required: ["label"]
So in the wire schema a prop with .default is marked required, while componentProps reports it optional and the playground prompt (input mode) agrees with componentProps. Two surfaces describing the same component disagree.
On the merits io: 'input' looks right: these are props a MODEL supplies, so the schema should describe what may be sent, not what comes back. But I have not changed it, for two reasons. convertZodToJsonSchema in tool-executor.ts is shared with the tool-definition path, so switching its mode would silently alter every tool's wire schema — that needs to be a parameter, not a global flip. And the API does not accept this plugin id yet (DEV-771/772), so nothing can validate which mode it actually wants; guessing now risks locking in the wrong one before the endpoint exists.
Your framing was "worth confirming which mode the API expects before DEV-772" — that is exactly right, and it is a question for whoever owns the endpoint. Flagging it in the PR summary too.
![perry-the-pr-reviewer[bot]](https://avatars.githubusercontent.com/in/3222961?s=60&v=4){kind=small}
This comment was marked as resolved.
This comment was marked as resolved.
Sorry, something went wrong.
serializeExpr emitted object keys raw, so a key with spaces, quotes,
punctuation, or a leading digit produced source the parser rejects. Keys
come from arbitrary tool-authored objects via toExpr, so they cannot be
assumed to be identifiers. The grammar already accepts a quoted key
(parseObject branches on '"'), so quoting the rest round-trips.
String(NaN)/String(Infinity) also emitted bare identifiers, which parse
back as refs to undefined names. JSON resolves the same hole as null; do
that rather than emit source that cannot round-trip.
fix(playground): describe enum props by their values, not "string"
describeSchema returned on json.type before checking json.enum, but an
enum serializes as {type: 'string', enum: [...]} — so every enum prop was
described to the model as a plain string and it never saw which values
are legal for Badge.tone, Stack.direction, Button.variant.
docs(changeset): add the required minor changeset for the OpenUI exports
~30 new exports plus ModelResult.getUiStream and the toUIOutput tool
option had no changeset, which the public-api-examples skill requires.
The example is compile-checked against the real signatures.
The gate's no_god_files rule is fan-out > 15, not file size: model-result.ts sat at exactly 15 outbound edges and this PR's ./openui/ui-stream.js import made it 16. Verified by removing that one import — the violation disappears. Re-exported UiStreamEvent and translateUiEvent from stream-transformers.js, which model-result.ts already depends on and which owns every other wire-event translation the loop performs, so no new edge is added. Complex functions were 9 -> 13, all four new here. Each is split along a seam it already had: - translateUiEvent: one function per wire event type (cc=18 -> under) - scanStatements: string-literal and bracket-depth state machines extracted - generate: the native path's per-variant event mapping extracted - renderCall: form controls and Table extracted to renderControl/renderTable Verified with sentrux 0.5.7, the version CI pins: God files 0 -> 0, complex functions back to 9 (the 9 remaining are all pre-existing on main and untouched), gate reports 'No degradation detected'. Behavior unchanged — full suite green, typecheck and lint clean.
The table of what each stream emits is the reference consumers use to pick one; getUiStream was absent.
…escript-agent-openui-module-libraryfragment-builders
cortex review —
|
![cortex-github-agent[bot]](https://avatars.githubusercontent.com/u/139423088?s=60&v=4){kind=small}
This comment was marked as resolved.
This comment was marked as resolved.
Sorry, something went wrong.
LukasParke
added
the
cortex-keep-updated
Two findings from cortex's review pass. XSS: the diagnostics panel escaped `source` but interpolated `message` and `line` raw into innerHTML. Every field there is model-controlled — `ParseFailure.message` is built from the offending source line, and in native mode diagnostics arrive verbatim off the wire — so a model could inject markup by emitting a crafted statement. All three fields are now escaped. A11y: rendered Input/Select carried no accessible name, so a screen reader announced an unlabelled field. Both signatures already have a `name` prop that was going unused for labelling; it now sets aria-label (and the real `name` attribute), falling back to the placeholder for Input.
There was a problem hiding this comment.
cortex panel verdict: comment — details in the consolidated review comment.
- escape diagnostic messages and history model names before innerHTML (XSS)
- bind playground server to 127.0.0.1 (API-key-backed endpoint)
- require trailing separator in static-file public-root prefix check
- validate uiRef/uiState/uiBuiltin names as identifiers at construction
- aria-label form controls from their name prop; progressbar ARIA + text %
- aria-live status/diagnostics regions; error frames no longer overwritten
by the green 'done' status
- rename toUIOutput -> toUiOutput (match Ui casing convention pre-release)
- warn (tool name + call id) when toUiOutput throws instead of catch {}
- collect toUiOutput broadcasts and await as one batch off the follow-up
critical path
- sticky regexes + charCode skipWs in the playground parser (was O(n^2))
- memoize openui(library) wire shape per library (WeakMap)
- drop the playground's no-op build script / outDir
…yfragment-builders' of https://github.com/OpenRouterTeam/typescript-agent into lukeparke/dev-773-typescript-agent-openui-module-libraryfragment-builders-2 # Conflicts: # packages/openui-playground/public/app.js
There was a problem hiding this comment.
cortex panel verdict: approve — details in the consolidated review comment.
…dule-libraryfragment-builders
|
|
cortex-github-agent
Bot
added
the
cortex-merge-conflict
|
|
|
|
|
|
|
|
|
|
3 similar comments
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
3 similar comments
|
|
|
|
|
|
1 participant
Summary
The Agent-SDK half of the OpenUI backport from Noetic (DEV-773, part of the DEV-765 umbrella; spec: DEV-764 RFC).
Deliberately thin per the spec: the API owns parsing/prompting/validation; the SDK ships builders, the tool render surface, and stream access.
New
src/lib/openui/moduledefineComponent/createLibrary— component vocabulary from Zod prop schemas; prop declaration order is normative (positional args in OpenUI Lang map by order)fragment(library)+uiRef/uiState/uiBuiltin— typed constructors for tool-authored UI; literal props validated at construction timeopenui(library)— produces the wire-shaped{id:'openui', library, dialect}plugin preference (Zod → JSON Schema);pluginsalready type-flows throughCallModelInput, so nocallModelsignature changetranslateUiEvent+ UI stream event modeltool()render surfacetoUIOutputsibling oftoModelOutputon regular/generator/HITL toolstool.ui_fragmentstream event (render-only — never sent to the model; throwingtoUIOutputdegrades to no-fragment)ModelResult.getUiStream()response.openui.statement/fragment/documentwire eventsUnknowncatch-all; translation reads the raw payload, so the stream works before and after the SDK regen (DEV-772)Deferred (documented in the ticket)
uiSubmitted()/uiInteracted()/uiToAssistant()stop predicates — need interaction events that only exist once Phase-3 surface state (DEV-774) landsTest plan
openui.test.ts,openui-stream.test.ts): serialization, library ordering/validation, fragment builder, plugin wire shape, event translation (incl. Unknown encoding),getUiStreamfast path,broadcastUiFragmentsuccess/skip/throw paths🤖 Generated with Claude Code