recommend times out when driven from a local agent to the MCP server

[FlorianGerdes]

TL;DR

When we run dot-ai as a remote MCP server (behind a load balancer) and drive the recommend tool from a local agent, longer recommend queries just… die. The client sits there, the progress bar freezes, and eventually we get a timeout — even though the server actually finishes the work fine a bit later.

We tracked it down to two things happening at the same time:

1. recommend goes quiet for a long time. During the AI-heavy phases there are stretches where nothing is sent back to the client for well over a minute.
2. The load balancer in front of the server has a short idle timeout. No bytes on the connection for X seconds → the LB assumes it's dead and closes it. The final response never makes it back to the client.

So it's not really a bug in the AI logic — the recommendation completes.

I wanted to open this up as a discussion because we've already hit it in our downstream fork, tried a few different fixes, and I suspect it overlaps with some existing PRDs. I'm not an expert on the internals here, so I'd love a sanity check on which direction makes the most sense before anything turns into a PR.

---

Initial observation

The first symptom was: running recommend from a local agent against the hosted server, longer queries would just fail with a timeout on the client side. No partial output, no hint about where it got stuck — the client hung for a while and then errored out.

Looking at the server logs, the recommendation actually ran to completion and produced a valid result. So from the server's point of view nothing was wrong; only the client ever saw a failure.

Observation after alb timout increase & implementing progress indicators

Once we increased the alb timeout & wired up notifications/progress (see the fork options below), we observed further timeouts. The client now showed step-by-step progress, and we could watch it get to step 4/5 ("Generating configuration options…") and then freeze there. It never advanced to step 5.

A concrete example from our environment: a recommend call took roughly 150 seconds end to end on the server, and the long silent stretch sat right after step 4. Our load balancer is an AWS ALB with a default idle timeout of 60 seconds, which we increased to 120. No bytes for 120s → the ALB closes the connection → the final response never reaches the client, which is exactly the freeze-at-step-4 we were seeing.

I don't think the exact numbers matter much — the shape of the problem is general: long silent gap + any proxy/LB with an idle timeout = dropped connection.

---

Deep dive 1: why does it go silent?

From what I can tell, the AI generation uses a blocking call (generateText), and a single recommend can chain several of these together (pattern search, ranking, then a loop that generates configuration questions). While those are running, nothing is streamed back to the client. The client only hears from the server again once a whole phase is done.

So there are "quiet windows" that scale with how much work the AI has to do. On a fast, simple intent you never notice. On a bigger one, a single quiet window is long enough to trip the timeout.

Deep dive 2: why doesn't the load balancer just wait?

Load balancers (and lots of reverse proxies) close connections that look idle, to avoid holding resources for dead clients. A long-running request that's genuinely still working, but not sending anything, looks identical to a hung connection from the outside.

You can bump the idle timeout up, and we did — but that only moves the goalpost. There's no timeout value that's correct for an operation whose length depends on the intent. The robust fix is to make sure something keeps flowing while the work is happening.

---

Related issues / PRDs I found

These all circle around the same area (long-running agentic operations and how the client hears about them), so I'm linking them for context rather than claiming they're duplicates:

• #455 – Evaluate streamText for long-running operations. This is the closest in spirit: same root observation (blocking generateText, up to ~20 iterations, no feedback until it's all done). The proposed direction there is streamText + onStepFinish for CLI/REST. Interestingly, that PRD explicitly sets MCP aside, but MCP does have a separate notifications/progress channel, which is exactly what makes the MCP case fixable after all. So I think our situation might be the MCP-shaped version of the same problem Evaluate streamText for long-running operations (REST/CLI) #455 describes.

• #460 – Add abort signal support for cancelling long-running AI operations. Complementary to what we did. It's the opposite direction: wiring MCP cancellation notifications into an AbortController on the vercel-provider tool loop.

• #461 / #457 (smoothStream UX / prepareStep for dynamic tool control) — seem somehow related.

If any of these is meant to already cover the MCP timeout case, point me at it and I'll happily fold our work into that instead.

---

What we tried in our fork (preliminary)

We ended up experimenting with three different levers.

Option 0 — just raise the load-balancer idle timeout (infra only)

The quickest fix: bump the ALB idle timeout (we went from the 60s default to a larger value). No code change.

• Pro: trivial, immediate relief.
• Con: doesn't actually fix anything. There's no "correct" timeout for an open-ended operation; a big enough query will still blow past it, and you're now also holding connections open longer for genuinely-dead clients. I consider this a band-aid, not a fix.

Option A — phase-based progress in the recommend pipeline

Thread a small progress callback through recommend and emit notifications/progress at the natural phase boundaries ("Searching patterns…", "Analyzing capabilities…", "Ranking…", "Generating options…", "Preparing…"). We also added a small periodic "heartbeat" wrapper around the two AI-heavy regions so that even within a long phase, something goes out every ~20s.

This is basically what we merged first internally.

• Pro: gives the user meaningful, human-readable progress; directly keeps the connection warm through the quiet windows.
• Con: it's specific to recommend. Every other long-running tool would need the same manual plumbing, and the phase markers are hand-placed.

Option B — generic per-step heartbeat for any agentic loop

The more general take. Instead of threading a callback through each tool's function signatures, bind the client's progressToken into an request context (via AsyncLocalStorage) when the request comes in, and hook into the provider's agentic loop so that every AI step emits a heartbeat. That makes it work for the other agentic tools too (impact analysis, query, operate, remediate), not just recommend, without touching each signature.

Key detail: it's a no-op when there's no progressToken (e.g. REST calls or clients that didn't ask for progress), and a failed notification never aborts the work.

• Pro: one place, covers all the long-running tools, no per-tool wiring.
• Con: the per-step heartbeats are "still alive" pings rather than semantically rich phase labels; less descriptive than Option A's messages.

---

What I'd like to discuss

• Is the MCP timeout case considered in scope for #455 (or somewhere else), or is it deliberately out of scope?
• Is notifications/progress an acceptable mechanism here, given tool-result streaming isn't available over MCP?
• Would you prefer any of the solutions or some combination or any other approach we haven't considered?

Happy to preliminary fork changes or broader fixes into a proper PR, but starting with a sanity check for now.

[vfarcic]

Thanks for the unusually well-researched writeup — the diagnosis is correct, and I've verified each piece against main (v1.25.0). Short version: yes, notifications/progress is the right mechanism; the fix is cheaper than you'd expect; and one of your two options wouldn't actually fix the bug you reported. Details below.

Confirming the diagnosis

• No progress signalling exists anywhere — grepping src/ and packages/ for notifications/progress, progressToken, and sendNotification returns zero hits.

• Blocking generateText throughout; no streamText, no onStepFinish.

• I found your "freeze at step 4". It's src/core/schema.ts:857-865:

  // Phase 3: Generate questions for each capability-based solution
  for (const solution of solutionResult.solutions) {
    solution.questions = await this.generateQuestionsWithAI(...);
  }

  Serial LLM round-trips, one per solution. With 3-5 solutions that's 3-5 full generations back-to-back with zero bytes out. That loop is most of your 150s.

Your three questions

Is the MCP timeout case in scope for #455? It should have been, and it wasn't — because #455 got it wrong. It says "MCP does not support streaming tool responses — so this won't help MCP clients" and scopes MCP out. Tool results are single-shot, true. But that isn't the same as MCP having no progress channel, and you're right to have spotted it. I'm rescoping #455 to REST/CLI token streaming and adding a correction note.

Is notifications/progress acceptable here? Yes. It's a base-protocol utility, out-of-band from the tool result — a separate JSON-RPC notification correlated by an opaque token, with the tools/call response still arriving once at the end, unchanged. Better still: dot-ai already sets enableJsonResponse: false (src/interfaces/mcp.ts:856), so responses go out over SSE and the stream is already open for the duration of the call. Notifications flow down that same connection with no transport change — which is precisely what stops the ALB seeing an idle socket.

Which option would I prefer? Neither as written — a corrected version of B, with A layered on top. Reasoning below.

Why Option B as specified wouldn't fix your bug

This is the important one. Option B hooks the provider's agentic loop so every AI step emits a heartbeat. But recommend never enters that loop.

All of recommend's AI phases go through aiProvider.sendMessage(), which at src/core/providers/vercel-provider.ts:415 is a bare single-shot generateText — no steps to hook. toolLoop (vercel-provider.ts:587) is used only by query, remediate, operate, and impact-analysis. So a step-driven heartbeat would light up four tools and leave recommend exactly as broken as before.

I think you half-sensed this already — it's why Option A needed that periodic wrapper bolted onto the two AI-heavy regions. The conclusion I'd draw is that per-step is the wrong axis: the silence is wall-clock silence, not step silence, so the heartbeat has to be time-driven.

The direction I'd take

B′ as the foundation — a time-based heartbeat rather than a per-step one. Your instinct to bind it into AsyncLocalStorage instead of threading callbacks is right, and there's less to build than you'd think: that context already exists (src/interfaces/request-context.ts:16) and is already run around MCP requests (mcp.ts:776) — it just carries identity today. The one real gap is that registerMcpTool (mcp.ts:262-303) discards the SDK's extra argument, which is what carries sendNotification and _meta.progressToken. Capture it there, start an interval, clear it in a finally.

Being time-driven rather than step-driven also means it covers non-AI stalls for free — capability vector search, knowledge search, ArtifactHub fetches, kubectl discovery.

A as a thin layer on top — your phase labels, on the same channel. The heartbeat guarantees liveness; the labels make it legible. Not either/or.

Option 0 you already called correctly. Agreed, band-aid.

Two things to watch

The client timeout is a second, independent ceiling. This one will bite you. The MCP SDK defaults to DEFAULT_REQUEST_TIMEOUT_MSEC = 60000 and only extends it on progress if the caller sets resetTimeoutOnProgress — with maxTotalTimeout capping it regardless. So heartbeats fix the ALB unconditionally, but fix the client hang only for clients that opt in. Worth checking your client config before you conclude the fix didn't work.

Your REST callers get nothing from this. src/interfaces/rest-api.ts:872 is single-shot JSON behind a hard requestTimeout, no notification channel. Same ALB exposure, not fixable this way. That part genuinely is #455 territory.

Also worth doing, separately

That schema.ts:857 loop has no reason to be serial. Promise.all would cut the worst window by roughly N×. It attacks the duration rather than the silence, so it's orthogonal to all of the above and needs its own thinking about provider rate limits — but it's real, and it came out of your report.

Next steps

I've opened #705 to track the MCP heartbeat work, written up with the above. #455 is being rescoped to REST/CLI. #460 (cancellation) stays separate but sequences naturally after #705, since it wants the same request context.

If you're up for turning your fork into a PR, #705 is the target — that way you're not carrying it downstream. The open questions there (heartbeat interval, whether phase labels reset the timer, which clients need resetTimeoutOnProgress verified) are exactly where your production experience would help most, since you've actually run this behind a real ALB and I haven't.