SlimSnap Schema

The open JSON format for screenshots and screen recordings fed to AI agents.

Terminal-based AI agents such as Claude Code, Aider, Codex CLI, Cursor CLI, can't accept image input. To talk to them about a UI, you have to describe it in English. That is slow, and the agent still guesses where things are.

SlimSnap is a Mac and Windows app that turns any screenshot into a small JSON blob: OCR'd text, element bounding boxes, your annotations. Since 0.6.0 it does the same for a screen recording: one frame per screen you visited, the clicks and scrolls that led from one to the next, and what you said while recording, as captions. You paste it into the terminal like code. Your agent reads it like code.

This repository is the open, MIT-licensed specification of that JSON format. The desktop app that produces it is closed-source (free during launch, with a paid tier planned); the format itself is free for anyone to read, write, validate, or implement.

→ Get the app: slimsnap.ai

---

Quick example

{
  "schema_version": "2.0",
  "captured_at": "2026-07-09T18:17:46Z",
  "mode": "single",
  "capture": { "width_px": 1440, "height_px": 900 },
  "screen": { "title": "Login - Acme", "app": "Safari" },
  "frames": [
    {
      "index": 0,
      "kind": "spatial",
      "dims": { "width_px": 1440, "height_px": 900 },
      "position": { "scroll_y_px": 0 },
      "elements": [
        { "id": "e1", "type": "label",  "value": "Sign in to your account", "bbox": [0.36, 0.18, 0.28, 0.06] },
        { "id": "e2", "type": "input",  "value": "Email",                   "bbox": [0.36, 0.32, 0.28, 0.06] },
        { "id": "e3", "type": "button", "value": "Sign in",                 "bbox": [0.36, 0.54, 0.28, 0.07], "color": "#3B82F6" }
      ],
      "annotations": [
        { "id": "a1", "type": "arrow", "color": "#EF4444", "from": [0.85, 0.30], "to": [0.55, 0.57], "intent": "highlight" }
      ]
    }
  ],
  "estimated_tokens": 340
}

A typical 1440×900 screenshot is ~7,500 vision tokens. The same screen as SlimSnap JSON is ~600 text tokens. About 12 times fewer tokens, and it works anywhere text does.

---

Why a format, not just an OCR dump

OCR alone gives text with no idea where it sits on the screen. A vision model knows where things are but the agent cannot quote them. SlimSnap puts both in one layout the agent can work with:

• frames: the unit of content. A regular screenshot is one frame; a scrollable capture (a whole page, a long chat, an endless feed) is several frames stacked top to bottom. Each frame is one coherent coordinate space.
• elements: every visible UI primitive with a normalized bbox (so coordinates stay sane regardless of the image's pixel size).
• annotations: what the human pointed at, as structured intent (highlight, explain, action, question), not just pixels of a red arrow.
• screen: optional context (URL, window title, app name) that the model uses to disambiguate.
• Deterministic IDs: every element and annotation has a stable string ID, unique across all frames, so the agent can refer back to "e4" instead of guessing "the green button on the right."

So the agent stops guessing what is in the image. It cites element ids like e4, reads what your arrow pointed at, and changes that one button.

---

Schema

The formal JSON Schema (draft 2020-12) lives at schema/v2.1.json. Use it to validate any SlimSnap export. schema/v2.0.json is the same spec without recordings; the previous major version stays available at schema/v1.0.json with examples in examples/v1/.

Top level

Field	Type	Required	Notes
schema_version	string	yes	"2.1" for this spec ("2.0" without recordings).
captured_at	string	yes	ISO-8601 timestamp.
mode	string	yes	"single" (one frame), "scroll" (scrollable capture) or "video" (recording).
capture	object	yes	{ width_px, height_px } of the WHOLE capture, all frames stacked.
screen	object	no	{ title?, app?, url? }. Where the capture came from (frame 0 for a recording).
frames	array	yes	The content, top to bottom or first to last. At least one frame.
media	object	video	{ type, file, duration_ms, fps }. The movie the frames came from.
interactions	array	video	Clicks and scrolls that moved the screen on, with the frame before and after.
transcript	array	video	[{ t0, t1, text }]. Everything said while recording, in time order.
ai_enrichment	object	no	Reserved for LLM-generated metadata. Absent when not enriched.
estimated_tokens	integer	yes	Approximate token count if fed to an LLM. For routing decisions.

Frame

{
  index:    number;                      // 0-based, top to bottom
  kind:     "spatial" | "temporal";      // spatial frames stack vertically, temporal frames follow in time
  dims:     { width_px, height_px };     // THIS frame's pixel size
  position: { scroll_y_px: number }      // spatial: top edge's offset inside the whole capture
          | { t_ms: number };            // temporal: ms from the start of the recording
  images?:  [{ file, offset_y_px, height_px }];  // present only when saved to disk
  elements:    Element[];                // coordinates normalized 0..1 within THIS frame
  annotations: Annotation[];             // same coordinate space
  unchanged?:  string[];                 // temporal: ids of earlier elements still on screen, not repeated
  screen?:     { title?, app?, url? };   // temporal: where this frame was taken, when it changed
}

Why frames: a 9,000px-tall scroll capture is useless as one coordinate space (a text line rounds to a 0.001-high bbox) and exceeds most canvas/vision limits as one image. Frames keep every capture, however long, in human-scale pages, each internally consistent. To get absolute pixel position of anything: y_px = position.scroll_y_px + bbox_y * dims.height_px.

Element

{
  id:    string;            // stable identifier, e.g. "e1"
  type:  "text" | "button" | "input" | "link" | "image" | "label" | "unknown";
  value: string | object;   // OCR text, or richer structured value
  bbox:  [x, y, w, h];      // normalized 0..1 floats
  color?: string;           // average RGB hex of the element region, e.g. "#3B82F6"
}

Annotation

{
  id:    string;            // "a1"
  type:  "arrow" | "rectangle" | "highlight" | "callout" | "note";
  color: string;            // hex, e.g. "#EF4444"

  // Geometry: only the relevant fields per type are set:
  from?:     [x, y];        // arrow start
  to?:       [x, y];        // arrow end
  position?: [x, y];        // single point (notes)
  bbox?:     [x, y, w, h];  // rectangle / highlight / callout

  text?:       string;      // callout / note copy
  target_ref?: string;      // ID of the element this points at
  intent?:     "highlight" | "explain" | "action" | "question";
  source?:     "voice";     // said aloud while recording, transcribed on device; absent when drawn
}

All coordinates are normalized 0..1 floats relative to the FRAME's dimensions (frames[].dims), so the JSON survives image resizing and rescaling. Element and annotation IDs are unique across the whole capture, so a target_ref near a frame boundary may point into the adjacent frame.

Recordings (mode: "video", since 2.1)

A recording is not a video file an agent has to watch. It is a list of the screens the user visited, what they did to get from one to the next, and what they said about each. Full example: examples/video-recording.json.

• One temporal frame per screen. A new frame is cut when the screen changes enough (a page load, a modal, a scroll past one viewport). Frames carry position.t_ms, their own screen when the app, title or URL changed, and unchanged, the ids of elements from earlier frames that are still there so the sidebar is not repeated six times.
• Captions are annotations with source: "voice". Each sentence lands on the frame it was said on, at the place the cursor was, with target_ref resolved to the element under it when there is one. So "this button is wrong" arrives as a callout pointing at el_014.
• Drawn annotations work like in a screenshot and sit on the frame they were drawn on. In the app they stay visible until the next click or scroll in the recorded app, then fade; the JSON keeps them on their frame.
• Interactions are the edges between frames:

{
  id:            string;              // "ix_001"
  t_ms:          number;
  source:        "click" | "scroll" | "screen";   // what was observed; "screen" = the screen changed, nothing watched the mouse
  point?:        [x, y];              // the click, on `frame`
  anchor?:       [x, y];              // where to point at this interaction on `frame`
  frame:         number;              // the screen it happened on
  result_frame:  number;              // the screen it produced
  target_ref?:   string;              // element acted on
  target_source?: "click" | "annotation";  // from the click position, or inferred from the annotation drawn just before
  target?:       { role?, kind?, label? }; // what the OS said the control was (needs Accessibility)
  effect:        "navigation" | "scroll" | "reveal" | "update";
  scrolled?:     number;              // scroll only: fraction of the viewport, positive downward
  change:        number;              // how much of the screen moved, 0..1
  changed?:      [x, y, w, h];        // where
}

• The transcript repeats every sentence with its start and end time, so an agent can read the narration straight through, and media points at the movie for a human who wants to watch.

To replay a recording: walk interactions in order, read frames[frame] for the state, target_ref for what was acted on, frames[result_frame] for what came of it, and the voice annotations on each frame for why. An agent that follows a bug report this way knows the exact element and the exact page, not "somewhere in checkout".

---

Validate

Any JSON Schema 2020-12 validator works. With Ajv:

npm install -g ajv-cli ajv-formats
ajv validate -s schema/v2.1.json -d examples/video-recording.json --spec=draft2020 -c ajv-formats
ajv validate -s schema/v2.0.json -d examples/annotated-screenshot.json --spec=draft2020 -c ajv-formats
ajv validate -s schema/v2.0.json -d examples/scroll-capture.json --spec=draft2020 -c ajv-formats

Should print ... valid for each.

---

Versioning

This spec follows semantic versioning at the field level:

• Patch (1.0.x): clarifications, fixed typos, expanded enums. Backward compatible.
• Minor (1.x): new optional fields. Backward compatible.
• Major (2.x): breaking changes (renamed/removed fields, type changes).

The schema_version field is a string and may be a major.minor pair (e.g. "2.1"). Producers should emit the highest version they fully implement.

v2.0 → v2.1: additive. New mode: "video", kind: "temporal" frames with position.t_ms, per-frame screen and unchanged, top-level media, interactions and transcript, and source: "voice" on annotations. A 2.0 consumer that ignores unknown fields reads a recording as a set of frames with annotations, which is still useful.

v1 → v2: v2's breaking change is the frame container. v1's top-level image / elements / annotations became capture + frames[].elements / frames[].annotations. A v1 document reads as a v2 document with one implicit frame: mode: "single", dims = image dims, position.scroll_y_px: 0. Consumers that want to accept both should branch on the presence of frames.

---

Use it

If you're building anything that takes a screenshot and feeds an agent, you're welcome to read or write this format directly. No payment, no attribution required. The desktop app at slimsnap.ai is the easiest producer; nothing stops you from writing your own.

A few obvious adjacencies if you want to pick up the thread:

• A browser extension that emits SlimSnap JSON from a webpage selection.
• A Linux producer using the platform OCR.
• A CLI that turns a saved PNG + manual annotations into SlimSnap JSON for batch workflows.
• A Claude Code / Aider slash-command that ingests SlimSnap JSON files directly.

Pull requests welcome on the spec itself, for new element types, new annotation intent values, new top-level fields. Keep them additive (minor version) where possible.

---

License

MIT. Use it however you want.

---

Built by @bickov · spec lives at github.com/bickov/slimsnap-schema · the app at slimsnap.ai