[tschuehly]

Draft. Blocked on two embabel-agent PRs:

• embabel-agent#2172 stops sending sampling parameters (temperature, top_p) that Claude 5 models reject.
• embabel-agent#2173 adds Claude Haiku 5.5 (CLAUDE_HAIKU_5_5) to the model catalog.

Without #2173 the examples cannot start on claude-haiku-5-5. Stacked on #141 (retired-model replacements), which touches the same lines.

Moves the example calls to the models a targeted benchmark supports, with effort set explicitly.

Mapping

Call	Before	After
Default model (both languages)	gpt-4.1-mini	Claude Haiku 5.5, low effort per call
cheapest role (both)	gpt-6-luna	Claude Haiku 5.5, low effort (Researcher.categorize)
Researcher critic (critiqueMergedReport)	gpt-4.1	Claude Haiku 5.5, medium effort
Researcher merge (mergeReports)	gpt-4.1 via criticModeName	gpt-4.1 via mergeModelName, which was declared but unused
Java FactChecker extraction	each of fact-checker.models	new fact-checker.extraction-models = Sonnet 5.5, low
Java FactChecker consolidation and reconciliation (deduplication-model)	gpt-4.1	Sonnet 5.5, low
Kotlin FactChecker extraction	Haiku 4.5 twice (temperature .3 and .0)	Sonnet 5.5, low and medium effort
Kotlin FactChecker merge	default model	Sonnet 5.5, low
Java fact verification (fact-checker.models), best, writing, research, auto-selected calls	unchanged	unchanged

How effort is set: Effort.LOW / Effort.MEDIUM in examples-common are Thinking.withTokenBudget(512 / 1024). Since embabel-agent#2141, the Anthropic converter maps these budgets to adaptive effort low / medium on Claude 4.7+. Effort is set per call because the default-model path does not carry role hyperparameters. InjectedComponent's structured joke drops temperature(.8), which Claude 5 rejects.

Evidence: per class (mechanical / semantic / both pass, n per row)

Class	Model/effort	n	Mechanical	Semantic	Both	$/case
extraction	Haiku 4.5	90	90%	88%	88%	0.000476
extraction	Haiku 5.5 low	90	92%	91%	91%	0.000075
extraction	Haiku 5.5 medium	90	93%	93%	93%	0.000088
extraction	Sonnet 5.5 low	90	97%	97%	97%	0.001128
read-only tools	Haiku 4.5	15	80%	73%	73%	0.002776
read-only tools	Haiku 5.5 low	15	80%	80%	80%	0.000287
read-only tools	Haiku 5.5 medium	15	80%	73%	73%	0.000294
read-only tools	Sonnet 5.5 low	15	80%	67%	67%	0.004577
bounded summary	Haiku 4.5	15	87%	13%	13%	0.003353
bounded summary	Haiku 5.5 low	15	87%	40%	40%	0.000451
bounded summary	Haiku 5.5 medium	15	73%	40%	40%	0.000536
bounded summary	Sonnet 5.5 low	15	80%	67%	67%	0.006765
creative writing	Haiku 4.5	45	96%	82%	82%	0.000782
creative writing	Haiku 5.5 low	45	96%	87%	87%	0.000100
creative writing	Haiku 5.5 medium	45	96%	80%	80%	0.000151
creative writing	Sonnet 5.5 low	45	98%	89%	89%	0.002193
research critic	Haiku 4.5	15	80%	80%	80%	0.001052
research critic	Haiku 5.5 low	15	80%	80%	80%	0.000143
research critic	Haiku 5.5 medium	15	100%	100%	100%	0.000194
research critic	Sonnet 5.5 low	15	87%	67%	67%	0.002987
research critic	Sonnet 5.5 medium	15	93%	73%	73%	0.002967

12 call sites × 5 synthetic fixtures × 3 reps per model. Haiku 4.5 ran without thinking; the 5.5 models ran with adaptive thinking.

Why Sonnet for the fact checker: every Haiku setting extracted an opinion from opinion-only text (0/3 each), and merged "Bats are not blind" with "Bats are blind" (0/3 each). Sonnet passed both 3/3.

Limits

• Same-family judge: Opus 5.5 graded Haiku and Sonnet. It ran blind, but it is not independent ground truth; a cross-provider judge is still open.
• Support bot: no model is safe on the current SupportAgent prompt; all claimed a stolen card "is blocked". That site needs #142 before any model choice holds.
• Summary and jokes: the summary word limit, the invented currency and the joke-prompt injection are prompt problems, fixed separately in #143.
• n=15 per site and model: one miss is 7 points. The tools, summary and critic classes are one site each.
• Calls the benchmark did not cover now inherit Haiku 5.5 low through the default model: NewsDigest and MarketIntelligence research and writing, GoatWriter evaluator, horoscope news search.
• With the default on Claude, an OpenAI-only setup may lose its working default model (not verified). Reviewers: is that acceptable for the examples?
• Not changed here: java-agent-template and kotlin-agent-template (WriteAndReview story on Sonnet 5.5).

[tschuehly]

🤖 Rebased onto main at 881ed6f. The PR is now the single commit 10b0bd0. The two earlier model-retirement commits were dropped because #141 already landed them.

StarNewsFinder conflict with #148: kept both changes.

• Fix: make MCP fetch optional for the Java StarNewsFinder #148's optional MCP fetch and its news-catalog selection flow are unchanged.
• Map example calls to Claude Haiku 5.5 and Sonnet 5.5 with explicit effort #144's web-tool find_news_stories call no longer exists. Its low-effort mapping now applies to the default-model calls that replaced it: select_news_stories in StarNewsFinder and fetch_news_feed in NewsFeedService. Both use withThinking(Effort.LOW).
• The Sonnet 5.5 writeup call keeps Fix: make MCP fetch optional for the Java StarNewsFinder #148's timeout and default-model fallback.

Other conflicts:

• InjectedComponent: kept Harden activity summary and joke prompts against overruns, invented currency and injection #143's prompt hardening and added Effort.LOW to both joke calls.
• InjectedComponentTest: kept Harden activity summary and joke prompts against overruns, invented currency and injection #143's FakeOperationContext tests and added a check that the joke call uses Effort.LOW.
• ActivitySummarizerTest: kept Harden activity summary and joke prompts against overruns, invented currency and injection #143's prompt test. The mock expectations now use withLlm(...) with Effort.LOW.

Tested: ./mvnw -pl examples-common,examples-java,examples-kotlin -am test passes, with 120 unit tests and 0 failures. I did not run LLM-backed or integration tests.