[Feature]: Mark a rate-limited model unavailable and fall back

[Fclem]

Problem

When the model bound to an agent profile stops on a provider rate limit or quota (for example a Codex/OpenCode usage-limit error), Kandev treats it as a session failure that needs a human, not as the model going away. The profile's configured fallback model is never used.

Current behavior, verified in this repository:

• Profile fallback is start-model-only. StartModelPolicy (FallbackModel, AutoFallback, RequireExactModel) in apps/backend/internal/agent/runtime/lifecycle/start_model.go applies FallbackModel only when the requested model is missing from the executor's advertised ACP catalog. No path applies it when an advertised model later rejects a prompt with a rate limit or quota error.
• Rate limits are classified, but not as "this model is unusable". routingerr defines quota_limited / rate_limited (apps/backend/internal/agent/runtime/routingerr/routingerr.go:38), and IsAvailabilityCode deliberately excludes transient rate limiting and quota from the "model unavailable, change the model" guidance.
• On a task session, a quota failure currently ends in a manual card. A reset time is surfaced but "never schedules a Kanban retry" (docs/specs/agents/requirements/agent-stall-recovery.md:124). Transient codes get a bounded same-model retry ladder (apps/backend/internal/orchestrator/event_handlers_transient.go); a hard quota failure does not.
• The machinery that would do this exists but is gated off. Provider/model-scoped degraded health with a retry time (apps/backend/internal/office/repository/sqlite/provider_health.go), reset waits (routingpolicy.ResetWaitPolicy, decision waiting_for_reset in apps/backend/internal/agent/runtime/routingpolicy/policy.go) and quota circuits (apps/backend/internal/agent/runtime/dynamic/circuit.go) all belong to dynamic agent routing, which is false in every shipped profile (profiles.yaml:75) and marked experimental/high risk (apps/backend/internal/runtimeflags/registry.go:147).

Net effect: one rate-limited provider stalls the task on a manual recovery card even when the profile already has a fallback model configured and the provider reported a reset time.

Proposed solution

For a model that stops on a rate limit or quota:

1. Mark that provider+model failed/unavailable durably, with the provider reset time when known, so later work that would use the same provider+model skips it until the reset passes or a bounded maximum wait expires.
2. If the agent profile has a fallback model configured and the executor advertises it, fall back immediately: continue the session on the fallback model without human action, and make the switch visible in the session.
3. If no fallback is configured, do not switch models. Hold the model unavailable until the end of the bounded wait derived from the reset time, then resume or retry the same model in the background.

Affected area

Agent lifecycle

Who needs this?

Individual developer

Target workflow

1. Create a task on the Kanban board and run it with a concrete agent profile that has a model (optionally a fallback model) configured.
2. The provider rate-limits or exhausts quota for that model mid-turn.
3. Today: the run ends, shows a recovery card naming the model and reset time, and waits for the user.
4. Wanted: the run continues on the fallback model, or resumes by itself after the known reset when no fallback exists.

Office autonomous sessions were not assessed by the reporter.

Alternatives considered

• Manual recovery card (today). Works, but needs a human and never uses the profile's fallback model.
• Dynamic agent routing (already implemented, flag-gated off). Provides cross-candidate fallback, reset waits, and provider/model circuits, but needs the experimental features.dynamicAgentRouting flag and a dynamic profile whose fallback is a different concrete profile, not the current profile's fallback model.
• AutoFallback on the profile. Session-start only, and it explicitly ignores the configured FallbackModel, so it cannot recover a rate-limited running model.

Acceptance criteria

• Given a running session whose profile model returns a rate-limit/quota failure, when the profile has an advertised fallback model, then Kandev switches that session to the fallback model and continues without human action, and the switch is recorded in the session.
• Given the same failure with no fallback configured, when the provider reports a reset inside the bounded maximum wait, then the model is marked unavailable until that reset and the session resumes or retries the same model in the background without a manual resume.
• Given an unknown reset, or one beyond the bounded maximum wait, then the run stops with the existing recovery surface and the model stays marked unavailable.
• The unavailable mark is scoped to provider+model, is used by later sessions on any profile that would use it, and clears when the reset passes.
• A strict profile (Require exact model on) never silently switches models; it fails visibly instead.
• Switching models stays visible and does not break the existing "one durable warning per selection decision" rule.
• No provider raw text beyond the existing sanitized, bounded diagnostic plus the reset time is persisted.
• Behavior when no rate limit occurs is unchanged.

Risks and constraints

• Contradicts two current documented rules: the Kanban "a reset time never schedules a Kanban retry" rule (agent-stall-recovery.md:124) and the "No silent model fallback" contract (docs/specs/agents/requirements/no-silent-model-fallback.md). A fallback must stay visible and must not weaken Require exact model, so this needs a deliberate spec change, not only an implementation tweak.
• Should reuse the existing classification, health, and reset-wait machinery rather than adding a parallel mechanism.
• Provider reset times can be missing, wrong, or far in the future, so a maximum wait bound is required (the existing routing policy caps reset waits at 7 days, far too long for interactive work).
• Office autonomous runs were not evaluated and may need a separate decision.

References

• Related: Feature: policy-based automatic model routing with safe fallback #2216 policy-based automatic model routing with safe fallback (open); Pull-based flow for perishable agent capacity: how kandev addresses it today, and what first-class support would look like #1669, [Feature]: Display AI agent session usage (5-hour and weekly quota) in the UI #1149 (closed, background)
• Specs: docs/specs/agents/requirements/agent-stall-recovery.md, docs/specs/agents/requirements/no-silent-model-fallback.md, docs/specs/agents/requirements/dynamic-agent-routing.md
• Code: apps/backend/internal/agent/runtime/lifecycle/start_model.go, apps/backend/internal/agent/runtime/routingerr/routingerr.go:38, apps/backend/internal/agent/runtime/routingpolicy/policy.go, apps/backend/internal/agent/runtime/dynamic/circuit.go, apps/backend/internal/office/repository/sqlite/provider_health.go, profiles.yaml:75

Before submitting

• I checked existing issues and did not find this request already tracked.
• This request describes user value, not only an implementation detail.

[ClemDNL]

The existing proposal covers rate-limit/quota fallback and provider reset waits. Please also capture this specific handling requirement:

• For an LLM HTTP 429 with error.type = "rate_limit_error", use fail-over only when it is both configured and enabled for the affected agent. A configured-but-disabled fallback is not eligible.
• If no eligible fail-over is available, automatically retrigger the request after the provider's retry-after-ms delay instead of leaving the task waiting for manual recovery. Preserve the value as milliseconds when scheduling the wait; do not interpret it as seconds or round it to a different duration.
• Example reported response (request ID omitted): {"type":"error","error":{"type":"rate_limit_error","message":"This request would exceed your account's monthly spend limit. Please try again later."}} with retry-after-ms=274579000 (274,579,000 ms = 3 days, 4 hours, 16 minutes, 19 seconds).

This is user-reported behavior; I did not independently reproduce it. No request IDs, credentials, customer identifiers, or internal Sentry details are included.

[OS-Juho-Lahtinen]

Adding a Claude Code use case for item 3 of the proposal: no fallback, resume by itself after a known reset.

Hitting the Claude subscription session limit stopped three Kanban sessions at once (Internal error: You've hit your session limit · resets 11:10am (Europe/Helsinki)). One of them was a workflow Implement step that was auto-starting. All three stayed down after the reset passed and had to be resumed by hand. With several parallel agents this is the common case: every session on the account hits the limit within a minute or so, and all of them need a manual restart.

What I'd want from this feature:

• When the reset is known and inside the bounded wait, resume each stopped session after the reset, as acceptance criterion 2 describes, including a workflow step whose auto-start failed on the limit.
• A per-provider/account view would help too. All sessions on the same account share one reset, so marking the provider+model unavailable (proposal item 1) should also keep new auto-starts from launching and failing right away during the wait.

Prerequisite: for Claude, the classifier doesn't recognize this message today. It is classified as agent_runtime_error / phase.poststart.unknown, and its reset format isn't parsed, so none of the quota handling applies yet. Filed separately as #4131.