Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -564,6 +564,7 @@ Text completion, vision, embeddings, and sub-calls route through plugin-register
- **`toolTransport`** — `grammar` (GBNF on llama-server) vs `native_tools` (OpenAI `tools` / `tool_calls`). Resolved by `resolveActiveToolTransport` from `config.llm.toolTransport` (`auto` follows the active provider).
- **Name escape** — qualified tool names use `__` for dots (`os.fs.read` → `os__fs__read`) in [openai-tool-call-adapter.ts](src/llm/provider/openai/openai-tool-call-adapter.ts). `reply` / `finish` are synthetic OpenAI functions alongside registry tools.
- **Vendor presets** ([src/tui/providers/provider-presets.ts](src/tui/providers/provider-presets.ts)) — 19 named cloud/local endpoints (Anthropic, Groq, Moonshot, Perplexity, Qwen/DashScope, SambaNova, …) that all resolve to the existing `openai-compatible` kind with `baseUrl` prefilled. Adding a vendor is a preset entry, not a provider kind. The one documented exception is `subscription-cli` — a subprocess backend has no baseUrl, no key and no HTTP path, so a preset cannot express it; within that kind the preset philosophy re-applies one level down (a new vendor CLI is a descriptor entry, never a new kind). Vendors that do not authenticate with `Authorization: Bearer` set `apiKeyHeader` (Anthropic: `x-api-key`) plus any mandatory static `headers` (Anthropic: `anthropic-version`); both are copied onto the saved config entry by [providers-wizard-build-entry.ts](src/tui/providers/providers-wizard-build-entry.ts) and applied to **both** request paths by the single [openai-auth-headers.ts](src/llm/provider/openai/openai-auth-headers.ts) builder, so discovery and chat cannot disagree. The bar for a new entry: probe `<baseUrl>/v1/models` **with the headers the preset will actually send** and get either 200 with a `data` array, or a 401/403 that rejects the *credential* — a 401 whose body names a header the preset does not send (`x-api-key header is required`, `Invalid bearer token` for what is an API key) is a **failing** probe, not a passing one. Either way the same host must answer 404 for a bogus sibling path; a gateway that rejects everything before routing proves nothing.
- **OpenRouter provider routing** — `llm.providers[].providerPreferences` is sent verbatim as the body's `provider` object by `buildOpenAiChatBody` (turns and sub-calls, streaming and unary) and `describeImageViaOpenAi` (vision). Only the `openrouter` factory forwards it: no other kind documents a `provider` field. It is set *before* the `extraBody` merge, so an explicit `extraBody.provider` — the old workaround — still wins. Deliberately **not** sent by `verifyProviderKey` (it probes the cheapest paid model, which a host pinned for the operator's model may not serve, and would misreport a good key as `model_unavailable`), the contract probe (built from wizard state, which carries no entry passthroughs — `extraBody` is absent there too), the catalog fetch (`GET /models`), or OpenRouter embeddings (a pin chosen for a chat model's hosts would strand an embedding model). Pinned by [openrouter-provider-routing.test.ts](src/llm/provider/openrouter/openrouter-provider-routing.test.ts) and [register-built-in-providers.test.ts](src/llm/provider/registry/register-built-in-providers.test.ts).
- **Bundled catalogs** — `OPENROUTER_MODELS_CATALOG` (split across `openrouter-frontier-chat-models.ts` / `openrouter-open-weight-chat-models.ts`) and `AIMLAPI_MODELS_CATALOG` are offline snapshots regenerated from each vendor's public `/models` endpoint; the shared row builders live in [model-catalog-entry.ts](src/llm/provider/model-catalog-entry.ts). Refresh = re-pull the endpoint, remap (`context_length`, `input_modalities` → vision, `supported_parameters` → tools, price × 1e6 → USD/1M) and update the date in each file header. `scoreChat` in the OpenRouter fetcher **ranks** vendors; it must not gate them — the Anthropic/Gemini exclusions it used to carry hid ~40 served models from the picker.
- **Model search** ([src/llm/provider/model-search.ts](src/llm/provider/model-search.ts)) — one ranked, multi-term scorer over model ids plus catalog metadata (vendor, `vision`/`text`, `tools`, `cache`, context shorthand like `1m`, `free`/`cheap`/`routed`). Tag matching is exact equality, so a context window is tagged three ways — as displayed (`1.0m`), floored to the whole unit (`1m`, the bucket a window falls in rather than a `>=` filter: 1_310_720 answers to both `1m` and `1.3m`, a 2M window only to `2m`), and, when the window is an exact multiple of 1024, in binary (131_072 answers to `128k`). Add a tag rather than changing [format-model-details.ts](src/llm/provider/format-model-details.ts): the display string is what the rows render. Terms are ANDed, matches are ranked (exact id > id prefix > vendor > word start > substring > subsequence) and equal ranks keep input order so the picker does not jitter per keystroke. Used by `filterModelIds` (TUI modal picker + Cloud pane) and by `atomic-agent models search`. Row rendering is shared through [format-model-details.ts](src/llm/provider/format-model-details.ts) — do not re-implement the price/context/capability strings in a frontend.

Expand Down
13 changes: 13 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -610,6 +610,19 @@ Turn it on only for a service that implements strict mode: one that does not wil

</details>

<details>
<summary><b>Choosing OpenRouter's upstream host</b> (<code>providerPreferences</code>)</summary>

OpenRouter serves most models from several hosts and picks one per request. To steer that — pin a host, forbid fallbacks, skip hosts that keep your data — set `providerPreferences` on an `openrouter` entry. It is sent unchanged as the request's `provider` routing object:

```json
"llm": { "providers": [{ "id": "openrouter", "kind": "openrouter", "providerPreferences": { "order": ["z-ai"], "allow_fallbacks": false } }] }
```

It applies to every chat completion the entry makes — turns, memory sub-calls and `vision.describe` — and other kinds ignore it. The pre-save key check does not send it: that check asks the cheapest paid model for one token, and a host pinned for your model may not serve that one. If you already set `extraBody.provider`, that keeps winning.

</details>

<details>
<summary><b>Configuration and secrets</b> (state dir, env vars, .env)</summary>

Expand Down
7 changes: 7 additions & 0 deletions src/config/config-schema.ts
Original file line number Diff line number Diff line change
Expand Up @@ -938,6 +938,13 @@ export interface AtomicAgentConfig {
supportsVision?: boolean;
requestTimeoutMs?: number;
promptCache?: "auto" | "off" | "explicit-markers";
/**
* OpenRouter provider routing (`order`, `only`, `ignore`,
* `allow_fallbacks`, `require_parameters`, `sort`,
* `data_collection`, …), sent verbatim as the chat body's
* `provider` object. Read by the `openrouter` kind only; an
* explicit `extraBody.provider` still wins.
*/
providerPreferences?: Record<string, unknown>;
/**
* Vendor-specific fields merged into the OpenAI-compatible chat
Expand Down
10 changes: 7 additions & 3 deletions src/config/llm-config.ts
Original file line number Diff line number Diff line change
Expand Up @@ -76,9 +76,13 @@ export type UserLlmProviderEntry = {
*/
promptCache?: "auto" | "off" | "explicit-markers";
/**
* Vendor routing preferences (e.g. OpenRouter's `provider` block).
* Same status as `promptCache`: carried through config, not yet read
* by any provider.
* OpenRouter provider routing — `order`, `only`, `ignore`,
* `allow_fallbacks`, `require_parameters`, `sort`, `data_collection`,
* … — sent verbatim as the chat body's `provider` object on every
* completion an `openrouter` entry makes: turns, sub-calls and vision.
* Other kinds ignore it. OpenRouter owns the vocabulary, so nothing
* here checks it beyond "an object". An explicit `extraBody.provider`
* still wins, since `extraBody` is merged last.
*/
providerPreferences?: Record<string, unknown>;
/**
Expand Down
29 changes: 29 additions & 0 deletions src/llm/provider/openai/openai-build-body.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -126,6 +126,35 @@ describe("buildOpenAiChatBody", () => {
expect(JSON.stringify(withUndefined)).toBe(JSON.stringify(withoutArg));
});

it("sends providerPreferences as the provider routing object", () => {
const preferences = { order: ["z-ai"], allow_fallbacks: false };
const args = [undefined, undefined, undefined, preferences] as const;
const unary = buildOpenAiChatBody({ prompt: "hi" }, "m", false, ...args);
const streamed = buildOpenAiChatBody({ prompt: "hi" }, "m", true, ...args);
expect(unary.provider).toEqual(preferences);
expect(streamed.provider).toEqual(preferences);
expect("provider" in buildOpenAiChatBody({ prompt: "hi" }, "m", false)).toBe(
false,
);
});

it("lets an explicit extraBody.provider win over providerPreferences", () => {
// `extraBody.provider` was the only way to route before
// `providerPreferences` was wired; merged last, it must keep working
// exactly as configured.
const override = { only: ["anthropic"] };
const body = buildOpenAiChatBody(
{ prompt: "hi" },
"m",
false,
{ provider: override },
undefined,
undefined,
{ order: ["z-ai"] },
);
expect(body.provider).toEqual(override);
});

it("does not let extraBody override reserved keys", () => {
const body = buildOpenAiChatBody(
{
Expand Down
6 changes: 6 additions & 0 deletions src/llm/provider/openai/openai-build-body.ts
Original file line number Diff line number Diff line change
Expand Up @@ -18,6 +18,7 @@ export function buildOpenAiChatBody(
extraBody?: Record<string, unknown>,
maxOutputTokens?: number,
strictTools?: boolean,
providerPreferences?: Record<string, unknown>,
): Record<string, unknown> {
const filtered = filterCloudCompletionRequest(request);
const body: Record<string, unknown> = {
Expand Down Expand Up @@ -114,6 +115,11 @@ export function buildOpenAiChatBody(
},
};
}
// OpenRouter provider routing (`order`, `only`, `allow_fallbacks`, …).
// Set before the passthrough on purpose: an explicit
// `extraBody.provider` is the older way to say the same thing, and it
// keeps winning. Absent, the body is byte-identical to what it was.
if (providerPreferences) body.provider = providerPreferences;
if (!extraBody) return body;
// Vendor passthrough. Merged last so it can reach fields this builder
// does not model, then reserved keys are restored on top.
Expand Down
5 changes: 5 additions & 0 deletions src/llm/provider/openai/openai-describe-image.ts
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,7 @@ export async function describeImageViaOpenAi(
defaultChatModel: string,
request: VisionRequest,
apiPathPrefix = "/v1",
providerPreferences?: Record<string, unknown>,
): Promise<VisionResult> {
const userContent: Array<
| { type: "image_url"; image_url: { url: string } }
Expand All @@ -32,6 +33,10 @@ export async function describeImageViaOpenAi(
max_tokens: request.maxTokens ?? 4096,
temperature: request.temperature ?? 0.1,
stream: false,
// The operator's images go wherever the turns go: routing is where
// `data_collection` / `only` / `ignore` live, and a describe call is
// as much a chat completion as a turn.
...(providerPreferences ? { provider: providerPreferences } : {}),
};
const start = Date.now();
const json = await openAiPostJson(
Expand Down
11 changes: 11 additions & 0 deletions src/llm/provider/openai/openai-provider.ts
Original file line number Diff line number Diff line change
Expand Up @@ -80,6 +80,12 @@ export interface OpenAiProviderOptions {
* Absent leaves the request body exactly as it was.
*/
strictTools?: boolean;
/**
* OpenRouter provider routing, sent as the body's `provider` object on
* every chat completion this client makes — turns, sub-calls, vision.
* Only the `openrouter` factory wires it; `extraBody.provider` wins.
*/
providerPreferences?: Record<string, unknown>;
/**
* Sink for the credit-limit retry warning (`plan-credit-limit-retry.ts`).
* Wired from the provider factory context so the notice lands wherever
Expand All @@ -103,6 +109,7 @@ export class OpenAiProvider implements LlmProvider {
private readonly extraBody: Record<string, unknown> | undefined;
private readonly maxOutputTokens: number | undefined;
private readonly strictTools: boolean;
private readonly providerPreferences: Record<string, unknown> | undefined;

constructor(options: OpenAiProviderOptions) {
this.id = options.id;
Expand Down Expand Up @@ -135,6 +142,7 @@ export class OpenAiProvider implements LlmProvider {
this.extraBody = options.extraBody;
this.maxOutputTokens = options.maxOutputTokens;
this.strictTools = options.strictTools ?? false;
this.providerPreferences = options.providerPreferences;
this.http = {
baseUrl: normalizeOpenAiBaseUrl(options.baseUrl),
apiKey: options.apiKey,
Expand All @@ -155,6 +163,7 @@ export class OpenAiProvider implements LlmProvider {
this.extraBody,
this.maxOutputTokens,
this.strictTools,
this.providerPreferences,
);
const json = await openAiPostJson(
this.http,
Expand All @@ -179,6 +188,7 @@ export class OpenAiProvider implements LlmProvider {
this.extraBody,
this.maxOutputTokens,
this.strictTools,
this.providerPreferences,
);
const path = `${this.apiPathPrefix}/chat/completions`;
let accumulated = "";
Expand Down Expand Up @@ -379,6 +389,7 @@ export class OpenAiProvider implements LlmProvider {
this.defaultChatModel,
request,
this.apiPathPrefix,
this.providerPreferences,
);
}

Expand Down
154 changes: 154 additions & 0 deletions src/llm/provider/openrouter/openrouter-provider-routing.test.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,154 @@
import { describe, expect, it, vi } from "vitest";

import type { CompletionRequest } from "../completion-types.js";
import { buildOpenAiChatBody } from "../openai/openai-build-body.js";
import {
OpenRouterProvider,
type OpenRouterProviderOptions,
} from "./openrouter-provider.js";

/**
* `providerPreferences` on the wire. It used to be parsed, validated and
* then dropped: an operator's `order` / `allow_fallbacks: false` never
* left the process, and OpenRouter kept routing wherever it liked.
* Asserted on the serialised request body, because that is the only
* place the omission was ever visible.
*/

const MODEL = "z-ai/glm-5.3-flash";
const PREFERENCES = { order: ["z-ai"], allow_fallbacks: false };

type Capture = { bodies: Record<string, unknown>[]; fetchImpl: typeof fetch };

function capture(reply: () => Response): Capture {
const bodies: Record<string, unknown>[] = [];
const fetchImpl = vi.fn(async (_url: string, init?: RequestInit) => {
bodies.push(JSON.parse(String(init?.body)) as Record<string, unknown>);
return reply();
});
return { bodies, fetchImpl: fetchImpl as unknown as typeof fetch };
}

const unaryReply = () =>
new Response(
JSON.stringify({
model: MODEL,
choices: [
{ message: { role: "assistant", content: "ok" }, finish_reason: "stop" },
],
}),
{ status: 200, headers: { "content-type": "application/json" } },
);

const streamReply = () =>
new Response(
[
{ model: MODEL, choices: [{ delta: { content: "ok" } }] },
{ model: MODEL, choices: [{ delta: {}, finish_reason: "stop" }] },
]
.map((frame) => `data: ${JSON.stringify(frame)}\n\n`)
.join("") + "data: [DONE]\n\n",
{ status: 200, headers: { "content-type": "text/event-stream" } },
);

function openRouter(
fetchImpl: typeof fetch,
extra: Partial<OpenRouterProviderOptions> = {},
): OpenRouterProvider {
return new OpenRouterProvider({
id: "openrouter",
apiKey: "test-key",
defaultChatModel: MODEL,
fetchImpl,
requestTimeoutMs: 5000,
...extra,
});
}

async function drain(
stream: AsyncGenerator<unknown, unknown, void>,
): Promise<void> {
for (;;) if ((await stream.next()).done) return;
}

const request: CompletionRequest = { prompt: "hi", maxTokens: 16 };

describe("OpenRouterProvider — providerPreferences", () => {
it("sends them as `provider` on a unary completion", async () => {
const { bodies, fetchImpl } = capture(unaryReply);
await openRouter(fetchImpl, { providerPreferences: PREFERENCES }).complete(
request,
);
expect(bodies[0]?.provider).toEqual(PREFERENCES);
});

it("sends them as `provider` on a streamed completion", async () => {
const { bodies, fetchImpl } = capture(streamReply);
const provider = openRouter(fetchImpl, {
providerPreferences: PREFERENCES,
});
await drain(provider.completeStream(request));
expect(bodies[0]?.stream).toBe(true);
expect(bodies[0]?.provider).toEqual(PREFERENCES);
});

it("sends them on a structured-output sub-call too", async () => {
// The reported case: a `response_format` sub-call that the pinned
// host cannot serve kept succeeding, because it was routed elsewhere.
const { bodies, fetchImpl } = capture(unaryReply);
await openRouter(fetchImpl, { providerPreferences: PREFERENCES }).complete({
prompt: "rewrite",
maxTokens: 64,
responseFormat: { name: "rewrite", schema: { type: "object" } },
});
expect(bodies[0]?.response_format).toBeDefined();
expect(bodies[0]?.provider).toEqual(PREFERENCES);
});

it("sends them on a vision describe call", async () => {
const { bodies, fetchImpl } = capture(unaryReply);
await openRouter(fetchImpl, {
providerPreferences: PREFERENCES,
}).describeImage({
prompt: "describe",
images: [{ id: 1, bytes: new Uint8Array([1]), mimeType: "image/png" }],
});
expect(bodies[0]?.provider).toEqual(PREFERENCES);
});

it("lets an explicit extraBody.provider win, unary and streamed", async () => {
const override = { only: ["anthropic"] };
const options = {
providerPreferences: PREFERENCES,
extraBody: { provider: override },
};
const unary = capture(unaryReply);
await openRouter(unary.fetchImpl, options).complete(request);
const streamed = capture(streamReply);
await drain(openRouter(streamed.fetchImpl, options).completeStream(request));
expect(unary.bodies[0]?.provider).toEqual(override);
expect(streamed.bodies[0]?.provider).toEqual(override);
});

it("leaves every body exactly as before when none are configured", async () => {
const unary = capture(unaryReply);
const streamed = capture(streamReply);
const vision = capture(unaryReply);
await openRouter(unary.fetchImpl).complete(request);
await drain(openRouter(streamed.fetchImpl).completeStream(request));
await openRouter(vision.fetchImpl).describeImage({
prompt: "describe",
images: [{ id: 1, bytes: new Uint8Array([1]), mimeType: "image/png" }],
});
expect(unary.bodies[0]).not.toHaveProperty("provider");
expect(streamed.bodies[0]).not.toHaveProperty("provider");
// Byte-identical to the builder called with its pre-existing arity.
expect(JSON.stringify(unary.bodies[0])).toBe(
JSON.stringify(buildOpenAiChatBody(request, MODEL, false)),
);
expect(JSON.stringify(streamed.bodies[0])).toBe(
JSON.stringify(buildOpenAiChatBody(request, MODEL, true)),
);
expect(vision.bodies[0]).not.toHaveProperty("provider");
});
});
5 changes: 5 additions & 0 deletions src/llm/provider/registry/provider-types.ts
Original file line number Diff line number Diff line change
Expand Up @@ -39,6 +39,11 @@ export type LlmProviderConfigEntry = {
supportsVision?: boolean;
requestTimeoutMs?: number;
promptCache?: "auto" | "off" | "explicit-markers";
/**
* OpenRouter provider routing, sent as the chat body's `provider`
* object. Only the `openrouter` factory forwards it; an explicit
* `extraBody.provider` still wins (see `openai-build-body.ts`).
*/
providerPreferences?: Record<string, unknown>;
/**
* Vendor-specific fields merged into the OpenAI-compatible chat
Expand Down
Loading
Loading