release: v0.6.1 (#411–#421) - #422
Merged
Merged
Conversation
The provider fallback chain partitions breaker state by the request's session id, and every memory sub-call except the query rewriter already used a prefixed id (reflection:, link-gen:, vote:). The rewriter ran on the bare session id, i.e. on the turn's own partition. A provider that refuses the rewriter's response_format request (a 404 "No endpoints found" on a pinned OpenRouter endpoint, a 400 on Qwen) throws an advance-worthy error, and the first advance-worthy failure sets the partition's sticky override. The user's next main step then went to the next link, usually the auto-appended local llama-server; with no local model running it died with "fetch failed", parked in the outage wait, and the turn failed. The rewriter itself is fire-safe and folds to the raw query, so the only visible symptom was the turn after. Run the rewriter under rewriter:<sessionId>. Its trace and metrics stay on the real session id. Usage metering keys by the same id, so the rewriter's tokens now fall out of the per-turn message_sent totals the way reflection, link-gen and vote already do (the global cost accumulator still counts them).
…easoning models
Reflection, the link-generator and the vote-runner run fire-and-forget
after every turn. Their defaults (10 s for reflection, which the vote
runner reuses, and 8 s for the link-generator) were tuned against a
local llama-server. Hosted reasoning models answer the same requests
far more slowly. Median latency of byte-faithful sub-call requests on
OpenRouter:
reflection glm-5.3-flash 15.7 s, qwen3.6-plus 16.3 s,
kimi-k2.6 13.9 s, gemini-3.8-flash 4.7 s
link-generator kimi-k2.6 37.8 s
In a live 8-turn session glm-5.3-flash reflection timed out on 6 of 8
calls at the 10 s default (25.4 s median, 37.5 s max), so most turns
wrote no memory at all.
Both defaults become 60 s. The timers are unref'd and nothing waits on
the reflection chain, so a longer cap cannot hold a turn or a one-shot
process open.
Config v65 migrates existing installs: every config.json already
carries these fields, written by the schema rather than chosen, so a
pre-v65 file whose value equals the old default takes the new one. Any
other value is a deliberate pin and is kept, as is the old number on a
v65 file. Same shape as the v63 managed.parallel migration. The helper
lives in its own module because config-schema.ts is far past the
per-file line budget.
The agent loop refreshes memory context before the first step and again after every step, each time with the same user message. The rewriter decorator asked its runner on every refresh, so one turn sent the same rewrite request once per step. It sits on the hot path: every call blocks the step until it answers or times out. Against a hosted model that is slower than the cap, each step paid the full timeout and still recalled with the raw message. Measured on OpenRouter (median, byte-faithful rewriter requests): gemini-3.8-flash 4.2 s, glm-5.3-flash 4.2 s, kimi-k2.6 23.5 s. In live 8-turn sessions gemini-3.8-flash's rewriter timed out on 16 of 18 calls at the 3 s default and kimi-k2.6's on 16 of 16. The decorator now remembers the outcome per session, keyed by the user message and a digest of the history slice it sent, and reuses it for every later refresh with the same key, a timed-out or failed attempt (outcome "use the raw message") included. A slow provider costs a turn one timeout, not one per step. Only the latest key per session is kept, for at most 256 sessions. An attempt whose caller's signal aborted is not remembered: a cancelled turn says nothing about the provider and must not decide the next turn's identical retry. With one call per turn the rewriter can afford a realistic cap, so its default goes 3 s -> 10 s inside the same v65 migration as the reflection and link-generator timeouts: a pre-v65 file carrying the old 3000 takes 10000, any other value is kept as a pin. It stays well below the background sub-calls because it still blocks the turn.
…refuses structured outputs Memory sub-calls (query rewriter, link generator, vote runner, distill) send OpenAI Structured Outputs. Endpoints without the feature refuse the request: OpenRouter pinned to Z.AI's endpoint for z-ai/glm-5.3-flash answers 404 "No endpoints found" because its "Filter by Parameters" routing step drops that endpoint, and vendor APIs without json_schema answer 400/422 naming the field. The sub-call then failed on every run (the rewriter failed 18 of 18 times in a live session), and because every cloud OpenAiHttpError classifies as transport, each refusal also advanced the provider fallback chain. OpenAiProvider.complete now recognises that refusal narrowly and sends the same body once more without response_format. The prompts still ask for their text formats and every parser reads them when the reply is not JSON, so only decode enforcement is lost. The retry happens inside the provider, so a handled refusal never reaches runWithFallback. The provider and model pair is remembered for the process once the stripped send is accepted, later sub-calls skip the field up front, and the first time logs one warn line. Not treated as refusals: 401/402/403/429/5xx, timeouts, network failures, size rejections, and the "messages must contain the word json" 400, which is a prompt problem on an endpoint that does support the feature. Streamed requests never carry response_format and are unchanged.
Every memory sub-call wrapper in bootstrap (reflection, link generator, vote, query rewriter, distill) enforced its runner's timeout by racing the completion against the abort signal without handing that signal to llmComplete. The runner gave up while the HTTP request kept running: still billing on a cloud provider, still holding a llama-server slot. The five copies are replaced by abortableSubcall, which forwards the signal into the request and keeps the race as a backstop for a provider that ignores it. Each wrapper still sends exactly the fields it did. Forwarding alone was not enough, for two reasons found on the way: - OpenAI-compatible unary requests stopped listening to the signal once headers arrived (openAiFetch unlinks it because it also opens streams), so a provider that sends headers before the body could not be cancelled. openAiPostJson now reads the body through a reader the signal cancels. - An aborted llama-server request surfaces as a status-null LlamaServerError, which classifies as transport and makes shouldAdvance switch providers immediately. The unary fallback seam now rethrows an aborted request as the signal's reason, so it classifies as cancelled and never trips a breaker or flips the override, the rule completeStream already applied.
OpenAI Structured Outputs compiles a strict schema before the model
runs and refuses it unless every object lists every property in
`required`. The link-generator and vote schemas left `links` and
`votes` optional for the `none` branch, so every one of those calls on
an OpenAI model answered 400 ("Missing 'links'") and nothing was ever
linked or voted on that provider.
Both keys are now required and the abstain branch carries an empty
array. The parsers already read `none`, and an empty array under
either kind, as none; new tests pin that. A test discovers every
*-response-format.ts under src/ and checks each exported schema
against the strict rules (closed objects, every key required, at every
depth), so the next sub-call schema cannot ship the same mistake. The
rewriter and both distill schemas already complied.
…vider routing
`llm.providers[].providerPreferences` has been parsed and validated
since the config fix that taught the loader about it, but nothing ever
put it on the wire. An operator who pinned OpenRouter's upstream host —
`{ "order": ["z-ai"], "allow_fallbacks": false }` — kept being routed
to whichever host OpenRouter picked, and structured-output sub-calls
the pinned host cannot serve kept "working" because they ran somewhere
else. The same object under `extraBody.provider` was honoured, which is
how the gap was found.
The field is now threaded exactly like `extraBody` and `strictTools`:
entry -> `openrouter` factory -> `OpenAiProvider` -> the request body,
where it becomes the `provider` object on every chat completion the
entry makes: turns and sub-calls, streaming and unary, and
`vision.describe` (routing is where `data_collection`, `only` and
`ignore` live, and those matter most for the operator's images).
Only the `openrouter` factory forwards it. `provider` is OpenRouter's
field, not the OpenAI schema's, and no other kind documents it.
It is set before the `extraBody` merge, so an explicit
`extraBody.provider` — the workaround people already use — still wins.
With no preferences configured the body is byte-identical to before.
Deliberately not sent by the pre-save key check (it probes the cheapest
paid model, which a host pinned for the operator's model may not serve,
and would report a good key as `model_unavailable`), the contract probe
(built from wizard state that carries no entry passthroughs, `extraBody`
included), the catalog fetch (`GET /models`, no body), or OpenRouter
embeddings (a pin chosen for a chat model's hosts would strand an
embedding model).
Alibaba-served Qwen refuses any request with `response_format` unless
the messages contain the word "json" ("'messages' must contain the
word 'json' in some form"), and OpenAI documents the same rule for
JSON mode. The rewriter, vote and link-generator prompts were written
for the llama-server GBNF path and never say it, so on qwen3.6-plus
every one of those calls answered 400.
buildOpenAiChatBody now appends one short instruction when it attaches
`response_format` and the prompt does not already mention JSON. It is
the only place that changes: the llama-server /completion payload is
built elsewhere, so the reflection slot's prompt bytes and KV cache are
untouched; the main agent turn sends no `response_format`, and a
request with `tools` never gets one, so their bodies stay
byte-identical. Tests pin all three.
runWithFallback dropped each failed link's error on advance and rethrew only the last one. On the common chain [cloud provider, auto-appended llama-server], a cloud 404 (OpenRouter: "No endpoints found for ...") followed by a local server that is not running reached the operator as `Turn failed [transport]: fetch failed`, the trace error row said the same, and the cloud's own refusal was logged nowhere, not even at debug. The last link's error is still the one thrown, unmodified: it decides classification, fallover and the outage wait, and those match on its class, cause, status and an anchored `fetch failed`, so rewriting its message would reclassify a bare TypeError as `tool`. The failed links are kept in a WeakMap beside it instead, found through the cause chain, which survives the step executor's TransportError re-wrap: - the TUI, Telegram and Discord failure lines append `(after "<id>" failed: <reason>)`; a single-link failure renders byte-for-byte as before - the trace error row carries them as `fallbackFailures` and keeps `message` verbatim; `trace show` prints them - the chain logs every advance at warn with from, to, status and the capped reason the switch notice already shows A turn already on a sticky override never retries the primary, and every retry of a parked turn is such a call, so the chain also remembers the primary failure that put the partition there and runWithFallback carries it onto a failure of the override link. Without that the turn still ended on a bare `fetch failed` once the five-minute wait ran out. describeReason and the breaker/partition state move to their own modules so provider-fallback-chain.ts stays within 300 lines.
Reflection, link generation, voting and the query rewriter run fire-and-forget after a turn and fail without a word. On hosted reasoning models, live sessions saw reflection time out 6 of 8 calls, the rewriter 16 of 16, and the vote runner refuse its schema 8 of 8. Those outcomes reached logs, metrics and trace rows but never the operator, and an empty link graph means the consolidator never distils a lesson, which nobody notices from the chat. A pure tracker (src/memory/health) counts consecutive timeout/failed outcomes per session and sub-call. ok, none and skipped reset the streak; aborted is neutral, because the next turn aborts a stale reflection by design. At three in a row it returns one warning per (session, kind) for the runtime's lifetime. The text names the knob: the per-call timeout key for timeouts, or the sub-call's switch plus the last failure reason (one line, credential shapes masked, capped) for failures. It quotes no default values, since those are moving. Bootstrap feeds the tracker from the existing emitTrace hooks, after the per-call trace row is written and whether or not the session is traced, and from the vote runner's run() result, which now carries the failure reason. The rewriter's trace event carries one too. The warning is logged at warn and emitted as a memory_health_warning event on the sub-call's own session: the trace recorder writes a row, `trace show` prints it, issue reports keep it at errors level without the reason, and the TUI shows a warn-styled system notice plus a feed line for the session on screen. Telegram and Discord get nothing, matching the fallover announcement.
…est's provider routing
…mode and Qwen reject
… hosted reasoning models
…endpoint refuses structured outputs
…llback #417 moved complete()'s body build into a closure handed to sendWithStructuredOutputFallback; thread this.providerPreferences through that closure (after strictTools) so unary sub-calls keep OpenRouter provider routing on both the first send and the prompt-only retry.
#414 appends a JSON instruction to the prompt of any body that carries response_format; #417's prompt-only retry drops response_format, so its prompt is the caller's own. The retry test asserted the two bodies were identical apart from response_format; assert the JSON mention on the first send and the original prompt on the retry instead. No runtime change.
…ned ones The ### profile section had one bound: render every fact sorted by key, then cut the string at memory.profile.maxTokens. In a long-running store the cut landed mid-section on every turn, sliced the last fact mid-value and removed whatever sorted late, pinned consent and security facts included, with a bare "[truncated]" inside the prompt as the only trace (#407). profile_facts was also the one memory table without maxEntries. - Render pinned facts first, then contextual ones (key order inside each group), and pack the section a whole line at a time: a line that does not fit is skipped, the last line counts what was left out, and BuiltPrompt.profileClip carries rendered / dropped / pinnedDropped. - AgentLoop turns a clip into a warn log and a profile_clipped event (trace row, trace show line, runtime_info line in the TUI feed), once per session and again only when the number of pinned facts left out changes. The total moves with every message because contextual facts are keyword-gated, so keying on it would warn most turns. Ephemeral fusion-worker turns skip it. - memory.profile.maxEntries (default 500) caps active unpinned facts. A set() past it deletes the lowest-utility unpinned facts (vote_score, then updated_at, then id; never the row being written) inside the insert transaction, the same delete remove() does, so superseded history stays readable. Pinned facts are never counted or evicted. Evictions log counts and write a profile_facts_evicted trace row whose keys /report strips below the full level. The key is additive with a default, so the config version stays at 64, as agent.task, agent.providerWait and agent.conversationMaxPairs did.
…ent stop When grammy's bot.start() settled without a stop() we asked for, the channel released its lock and went `down` for good. The process kept running and looked healthy while every Telegram message went unanswered until someone restarted it. The Discord gateway in the same codebase has always reconnected. An unexpected stop now arms one unref'd retry of start() on the Discord gateway's full-jitter backoff. The backoff moves to src/channels/reconnect-backoff.ts and is re-exported from the Discord transport, so its import path, behaviour and tests are unchanged. While waiting, the channel reports `down` with "polling stopped: <reason> — reconnecting in Ns (attempt n)": a new ChannelState would ripple through the Integrations hub, the setup panel, the status lines and the sidecar for no gain. Each attempt is a warn line with the scrubbed reason, so `serve` leaves a post-mortem trail on stderr. No retry can fix a Bot API 401 (revoked token), 404 (malformed token) or 409 (another process polls the bot, and retrying would fight it), nor a lock another process holds. Those still end `down` with the plain reason. The attempt count starts over only after a full 30 s long-poll round of `up`, so a stop that recurs right after every start backs off to the cap instead of spinning. A first start() that fails stays `down`, as before. stop() disarms the timer and bumps a generation that a start() still awaiting Telegram checks before committing, so neither a waiting nor an in-flight retry can resurrect a stopped channel. Any other start() supersedes the waiting retry; restart() and setToken() treat a waiting channel as running. An unexpected stop also drops the per-session approval bindings: they pointed at the dead bot's bridge and would have posted keyboards whose clicks the reconnected bot ignores. Refs #409: the reconnect covers every non-fatal loop exit; a 409/401 exit stays terminal by design and is now logged with its reason, so the issue stays open until that reason is confirmed.
… a transient stop
… drop pinned ones # Conflicts: # src/cli/trace-formatter.test.ts # src/cli/trace-formatter.ts # src/tui/issue-report/trace-redaction.ts
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Release integration for v0.6.1: every open maintainer PR, merged into one branch so conflicts are resolved once and the release is built from exactly what was tested.
providerPreferencesis sent to OpenRouter as the request'sproviderroutingresponse_formatmention JSON (Qwen)response_formatwhen the endpoint refuses structured outputsmemory.profile.maxEntriescaps unpinned facts (fixes #407)Why one PR
Two reconciliations exist only on this branch and are lost if the member PRs are merged one by one:
03326294threads fix(llm): send providerPreferences to OpenRouter as the request's provider routing #413'sproviderPreferencesinto fix(llm): retry a sub-call without response_format when the endpoint refuses structured outputs #417's structured-output retry, so sub-call retries keep OpenRouter routing;8d980b1fupdates fix(llm): retry a sub-call without response_format when the endpoint refuses structured outputs #417's retry test for fix(memory): structured-output sub-calls that OpenAI strict mode and Qwen reject #414's JSON mention (the retry sends the caller's own prompt).#421's merge also resolved additive conflicts with #418 / #419 in the trace formatter, its tests and the
/reportredaction lists.How it was verified
npm run lintclean;npm run buildcleansrc/llm src/runtime(109 files / 1 317),src/memory src/config(65 / 962), tracing + cli + channel handlers + agent (48 / 777), targeted TUI (11 / 173), and after fix(telegram): reconnect the polling loop with backoff after a transient stop #420 / fix(memory): cap profile facts and never let the prompt clip drop pinned ones #421: channels (27 / 520), fix(memory): cap profile facts and never let the prompt clip drop pinned ones #421 areas (72 / 1 231),src/agent src/runtime(48 / 585), memory + memory tools + Discord (71 / 847) — all exit 0runexits 0 and writes config v65 with the new defaultsworkflow_dispatch,publish=false) on this branch succeeded; the signed darwin-arm64 binary boots and migrates config