Skip to content

replace gemma image reading with moondream - #222

Merged
deathbyknowledge merged 8 commits into
mainfrom
moondream-img2txt
Jul 26, 2026
Merged

replace gemma image reading with moondream#222
deathbyknowledge merged 8 commits into
mainfrom
moondream-img2txt

Conversation

@deathbyknowledge

Copy link
Copy Markdown
Owner

Summary

  • replace the configurable Gemma image reader with the sole Workers AI model @cf/moondream/moondream3.1-9B-A2B, with no fallback path
  • expand ai.image.read and img2txt across caption, query, OCR, point, and detect modes
  • default img2txt to caption while keeping query prompts caller-owned
  • add query reasoning/grounding, JSON/XML/Markdown/CSV structured output, schema checks, and mode-specific limits
  • stream caption, query, and OCR through response bodies with cancellation and timeout cleanup
  • preserve meaningful internal whitespace, remove old provider/model/input-format/prompt configuration, and simplify console settings
  • update native command discovery, protocol/reference docs, and the shipped GSV Manual skill

Companion user/agent manual update: deathbyknowledge/gsv-manual#1

Validation

  • gateway TypeScript check and full suite: 1325 passed, 5 skipped
  • public SDK: 49 passed
  • CLI/device Rust: formatting clean, 94 passed
  • web: typecheck and production build pass; 277 tests pass; the unchanged ChatTranscript.test.ts bootstrap failure on DOMPurify().addHook remains reproducible on origin/main
  • extension: check, 18 tests, and build pass
  • assembler: 52 passed
  • WhatsApp TypeScript and Discord/Telegram/test-adapter typechecks pass
  • ripgit worker build passes; its configured Vitest run contains no test files
  • focused bootstrap/config tests after the manual-skill update: 18 passed

@deathbyknowledge

Copy link
Copy Markdown
Owner Author

Live Workers AI testing found and fixed a response-shape mismatch in 54c08066:

  • non-streamed env.AI.run() responses wrap Moondream fields under { result: ... }; the inference boundary now unwraps that envelope for caption, query/OCR, point, and detect
  • streamed query/caption events carry cumulative text under chunk.answer / chunk.caption; the decoder now emits only each new suffix and ignores lifecycle/final aggregate events
  • regression mocks now match the live binding shape

Validation: gateway TypeScript passes; 92 focused tests pass; full gateway suite passes (1325 passed, 5 skipped). A fresh CI run is in progress.

@deathbyknowledge

Copy link
Copy Markdown
Owner Author

@codex review

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: a165e6b70d

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

schema: Record<string, unknown>,
path: string,
): void {
if (Array.isArray(schema.enum) && !schema.enum.some((item) => Object.is(item, value))) {

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Enforce the submitted JSON Schema

When callers use standard JSON Schema constraints outside this small hand-coded subset—for example, { "type": "number", "minimum": 10 }—a Moondream result of 5 is accepted and returned as structured; object and array enum values are also incorrectly rejected because Object.is compares references rather than JSON values. This makes the advertised schema check unreliable for downstream automation. Use a compliant validator, or explicitly reject unsupported keywords and implement JSON-value equality for enums.

AGENTS.md reference: AGENTS.md:L43-L43

Useful? React with 👍 / 👎.

@deathbyknowledge
deathbyknowledge merged commit 1cc4db3 into main Jul 26, 2026
7 checks passed
@deathbyknowledge
deathbyknowledge deleted the moondream-img2txt branch July 26, 2026 08:40
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant