Fresh install: capabilities never get scanned (flaky)

[FlorianGerdes]

Hello @vfarcic . Thank you for your quick and constructive response on the last discussion. It encourages me to keep engaged. I followed up with an PR and starting this discussion which describes another issue we faced in our setup.

TL;DR

On a fresh dot-ai-stack install, we repeatedly ended up with an empty capabilities collection — the capability scan just never populated it.

Digging in, it wasn't one bug but a small pile of them that line up badly on a fresh cluster:

1. The initial scan only runs once. The controller does its "compare cluster resources vs known capabilities and scan the missing ones" pass on startup / when the config resource is (re)created — but there's no periodic retry. If that one attempt happens before the vector backend is actually ready, it quietly fails and never runs again.
2. Failures get masked as "nothing to do." When the scan trigger fails because the backend isn't ready, the controller doesn't surface an error — the config still shows Ready with no lastError, which looks identical to a healthy, fully-scanned system.
3. The controller under-counts existing capabilities. Because of a response-parsing mismatch and a server-side list cap of 100, the controller sometimes reads far fewer capabilities than actually exist, which then drives it to re-scan everything over and over.

I'm opening this as a discussion because we've hit all three in our downstream fork and patched them. However I'm not sure which of these issues "hold true" for other environments or whether my analysis is even fully reasonable. Note this spans two repos — most of it is controller-side (dot-ai-controller), plus one change in the MCP server (dot-ai).

---

Initial observation

Fresh install of the stack, wait for everything to go green, then ask recommend for something — and it behaves as if the cluster has no capabilities at all. Checking the vector store directly, the capabilities collection was empty (0 documents), even though the cluster had a few hundred resource types that should have been scanned.

The CapabilityScanConfig resource showed up normally. There was no error nor a a previous event recorded on it, the controller pod was happily running with zero restarts, and the MCP server answered normally. So from every status surface we had, the system claimed to be fine — it just hadn't actually scanned anything.

The one reliable "fix" we found early was to restart the controller. After a restart the scan ran, the collection filled up, and recommend started working. That was the clue that the scan was tied to startup rather than to an ongoing "keep things in sync" loop.

---

Deep dive 1: how is the scan supposed to happen, and where does it fall down?

The scan is driven by a the dot-ai-controller. The design — laid out in the PRDs linked below — is roughly: on startup, list the cluster's resource types, list what's already in the capabilities index, diff them, and scan whatever's missing (a full scan when the index is empty, which is exactly the fresh-install case).

The trouble is that this diff-and-scan is bound to startup and to the config resource being (re)created. On a fresh install the config gets created once, which fires exactly one scan attempt. If the vector backend / embedding subsystem isn't functionally ready at that precise moment — and it often isn't, because the MCP server's readiness check doesn't actually verify the capability subsystem — that single attempt fails. There's no periodic "try again" loop, so the system just sits there empty until something external (a controller restart, a config change) happens to trigger another scan.

Deep dive 2: why didn't we see an error?

When the scan trigger failed because the backend wasn't ready, the failure wasn't surfaced. The backend returns an HTTP 200 with a "success: false" body (reachable but not functional) — and the controller interpreted that as "list returned 0 capabilities," not as "this call failed." So it happily concluded there was nothing wrong.

Deep dive 3: the re-scan storm (the opposite failure)

Root cause there was under-counting: the controller was reading far fewer capabilities than were actually stored — partly a response-shape parsing mismatch (an extra nesting level, plus matching on the wrong field), and partly a server-side list endpoint that silently capped results at 100. Either way the diff always thought a big chunk was "missing" and kept re-scanning. So the same subsystem can fail in two opposite directions — scan nothing, or scan everything endlessly — and both trace back to the controller not having an accurate picture of what's already there.

---

Related issues

Linking these for context:

• dot-ai #216 – PRD: Controller-Based Autonomous Capability Scanning (closed/completed). The umbrella PRD. Its own description says the controller should "check for existing capabilities on startup, trigger full scan if needed".

• dot-ai-controller #34 – PRD: Autonomous Capability Scanning Controller (closed/completed). Phase 2 of PRD: Controller-Based Autonomous Capability Scanning #216, the actual controller implementation. Its commit history explicitly notes "Remove InitialScanComplete status check (always reconcile on startup)" and "Full scan only when MCP is empty (fresh install)".

• dot-ai-controller #28 – Resource Visibility and Status Tracking (closed/completed). The sibling controller for resource sync already ships "initial sync & periodic resync … on startup and hourly" with a configurable resyncIntervalMinutes.

• dot-ai #51 – Capability Scanning Performance Optimization (closed, superseded by PRD: Controller-Based Autonomous Capability Scanning #216). Relevant to the re-scan storm we experienced.

---

Downstream fixes

Fix 1 — autonomous periodic resync (controller)

Add a resyncIntervalMinutes field to CapabilityScanConfig (default 60) and, on the "config unchanged" reconcile path, actually re-run the diff-and-scan on that interval instead of just idling. This makes the initial scan self-healing: if the first attempt fails because the backend wasn't ready, a later resync picks it up — no controller restart or CR recreation required. This deliberately mirrors what the ResourceSync controller in #28 already does.

• Pro: fixes the core "empty forever" case; small, matches an existing pattern in the repo.
• Con: picking a sensible default matters.

Fix 2 — surface MCP backend failures instead of masking them (controller)

Treat an operation-level failure (HTTP 200 but "success: false", e.g. vector backend not ready) as a real, retryable error rather than as "0 capabilities." That way a failed scan is retried and shows up as an error instead of silently looking like a healthy, fully-scanned system.

• Pro: kills the "everything looks fine but nothing happened" trap; makes the status trustworthy.
• Con: it just surfaces the problem.

Fix 3 — stop under-counting existing capabilities (controller + MCP server)

Two parts that together stop the re-scan storm:

• Controller: parse the actual (nested) shape of the MCP list response and match capabilities by their resourceName rather than by the internal UUID id, so the diff compares like with like.

• MCP server (dot-ai): raise the capabilities list cap from 100 to other limits (10000), since the controller needs the full list to compute an accurate diff. (This is the one change that lives in the dot-ai repo rather than the controller.)

• Pro: the controller finally sees the true count, so the diff converges to zero and the storm stops.

---

What I'd like to discuss

• Is autonomous periodic resync for capability scanning something you'd want upstream, or was startup-only + external retrigger a deliberate design choice I'm missing?
• Should a scan trigger that fails because the backend isn't ready be surfaced on the CR status / retried, rather than treated as an empty result?
• Is the 100-item cap on the capabilities list endpoint intentional? If so, what would be the side effect of raising the limit?

[vfarcic]

Thank you for this — it's an exceptionally good report, and you were right to be suspicious that it wasn't three unrelated bugs.

We traced all of it through the code. Your analysis holds, and the three symptoms collapse into one root cause: the controller's ManageOrgDataResponse struct (capabilityscan_mcp.go:43-59) doesn't match what the MCP server actually returns. It's wrong in three independent ways, and each one on its own is enough to break reconciliation.

Also — to get this out of the way — this is our bug, not an artifact of your environment. The root cause is traceable to our own PRD, so the uncertainty you expressed about whether your analysis generalizes is misplaced. It generalizes to every deployment. Details below.

What we confirmed

(a) The payload is one level deeper than the controller looks. The REST layer wraps tool output as {success, data:{result:<toolOutput>, ...}} (rest-api.ts:918), and the tool output is itself {success, operation, dataType, data:{capabilities, totalCount, ...}} (capability-operations.ts:113). So the real path is data.result.data.capabilities — which our own integration tests assert (tests/integration/tools/manage-org-data-capabilities.test.ts:157). The controller declares capabilities directly under data.result.

⇒ GetCapabilityIDs() always returns empty. GetTotalCount() always returns 0. Unconditionally, on every install.

(b) id is not the resource identity. It's a deterministic UUID — sha256("capability-" + resourceName) reformatted 8-4-4-4-12 (capabilities.ts:328-337). computeCapabilityDiff compares it against the Kind.group strings the controller builds from Discovery, so it could never match even with (a) fixed. The field you want is resourceName, already in the same payload — exactly as you proposed.

(c) resp.Success is the transport flag, not the operation result. manageOrgData catches every error and returns normally (organizational-data.ts:868-905), so the envelope's success: true only means "the handler didn't throw." For this endpoint it is effectively a constant. An unreachable Qdrant returns {success:false, error:{message:'Vector DB (Qdrant) connection required'}} at data.result under HTTP 200 with envelope success:true (capability-operations.ts:884-897).

One correction to your framing, in our favour: the inner flag does cleanly separate the two states. An absent collection returns inner success:true with an empty array (capability-operations.ts:903-913) — the honest answer on a fresh install — while an unreachable backend returns inner success:false. The signal was always there. We just never published it, and the controller never read it.

Why this produces exactly what you saw

"Empty forever, but green": TriggerScan gets 200 + inner success:false, so sendWithRetry treats it as success on the first attempt — the retry/backoff you configured never engages. The controller logs ✅ Targeted scan triggered successfully and calls updateLastScanTime, which sets LastScanTime and clears LastError (capabilityscan_controller.go:640-642). Ready, no error, recent scan time, empty collection.

Re-scan storm: the list always reads empty, so the diff puts every cluster resource type into toScan on every start, no matter how full the collection already is. Your under-counting diagnosis was right; the count is simply always zero.

Your point 1 also stands independently — performStartupReconciliation only runs when a config is newly added to activeConfigs, and the unchanged-config path early-returns at capabilityscan_controller.go:113-116. The reconciler wakes every 60 seconds and deliberately does nothing.

Why CI never caught it — and why this is on us

Our e2e mock MCP server hand-writes the response as data.result.capabilities with id values "Pod", "Service", "FakeStaleCRD.old.example.com" (test/e2e/mock-capability-scan-server.yaml:20-23), and the unit fixtures do the same (capabilityscan_mcp_test.go:167-184). Both encode the same two wrong assumptions as the production code, so our tests assert the bug.

And those assumptions came from PRD #34's API reference, which documented data.result.capabilities and elided the array contents as "capabilities": [...] — hiding the id/resourceName distinction entirely. The contract was written by hand from prose instead of captured from a real response, and everything downstream faithfully implemented the wrong spec.

Two things your analysis didn't cover

1. Fixing the parse arms a deletion path that has never run. This is the one we'd most want you to know before touching the code.

Because mcpSet is always empty today, toDelete is always empty (capabilityscan_controller.go:317-322) — the orphaned-capability deletion has never executed in production. Your Fix 3 is what turns it on.

That path is unguarded, and listAllResourceIDs deliberately tolerates partial discovery failure (capabilityscan_controller.go:415-422) and proceeds with an incomplete cluster list. So an aggregated APIService that's briefly unavailable — a metrics adapter mid-restart, a webhook backend flapping — omits every kind in that group from clusterSet, and every stored capability for those kinds gets deleted, then re-scanned on the next cycle. Expensive, and it flaps.

The same reasoning applies to the MCP side: any list the controller can't prove is complete must not drive deletions. So a completeness guard needs to land in the same change as the matching fix, not after it. Concretely: scan-when-incomplete is safe (scans are idempotent), delete-when-incomplete is not.

2. Scans are fire-and-forget, so "trigger accepted" is all you can learn. handleFireAndForgetScan saves a session and starts the scan without awaiting (organizational-data.ts:186-255). Even with (c) fixed, a successful trigger tells you nothing about whether the scan finished — a pod restart mid-scan, an AI provider outage, or per-resource inference failures all leave a partial collection under a green CR.

We think this is the actual reason it's flaky rather than deterministic. There's a progress operation for exactly this, and the controller never calls it.

Your three questions

1. Do we want autonomous periodic resync upstream? Yes. Startup-only wasn't a deliberate design choice you're missing — and your instinct to point at ResourceSyncConfig is exactly right; that's the precedent we'll mirror (resourcesyncconfig_types.go:27-33, resourcesync_controller.go:489-490, :1396). PRD #34's own commit note, "Remove InitialScanComplete status check (always reconcile on startup)," shows the intent was to re-derive from observed state, which a resync completes rather than contradicts. (InitialScanComplete is now dead weight — declared, never set, never read.)

⚠️ But the ordering in your write-up is backwards, and it matters. Resync amplifies whatever the diff concludes. Landing it while the diff still reads zero means a full-cluster AI re-scan every 60 minutes, forever — considerably worse than today's once-per-restart. Fix 3 has to come first.

2. Should a failed trigger be surfaced and retried? Yes — and the fix is checking data.result.success, which routes it into the backoff you already configured. Two additions: updateLastScanTime must stop clearing LastError, and LastScanTime shouldn't claim a scan happened when only a trigger was accepted (that needs the progress poll above).

3. Is the 100-item cap intentional? We don't think so — it reads as a display guard rather than an API limit. It sits next to clientInstructions telling an agent to "display capability list with IDs prominently visible," and the schema advertises "default: 100" rather than a maximum.

On side effects of raising it: cheaper than you'd expect, because the expensive part already happens unconditionally. getCapabilitiesCount() calls getAllData() with no limit (capability-vector-service.ts:214-218), resolving to limit ?? 10000 (base-vector-service.ts:312-316) — a full 10 000-document scroll with payloads on every list call, including limit: 1. The cap only trims what gets serialized, not what gets read. So the marginal cost is response size, not database work. (That unconditional scroll is a separate bug we're fixing while we're in there.)

Two refinements on your Fix 3, though:

• The controller only needs resourceName, so 10 000 full records — descriptions, capability arrays, printer columns — is a lot of payload for a set of identifiers. We're adding an identity-only projection alongside the raised ceiling.
• totalCount is not capped, so it's an accurate signal today. Comparing it against returnedCount is how the controller can detect truncation — which is what lets a fixed controller run safely against an unpatched server instead of needing both sides upgraded in lockstep.

One more, which you inferred correctly

Your remark that "the MCP server's readiness check doesn't actually verify the capability subsystem" is right, and worse than stated. /healthz is a hardcoded {status:'ok'} (mcp.ts:723-727), and the chart points both probes at it (charts/templates/deployment.yaml:265-278). The pod is Ready as soon as the listener binds.

Local embeddings make it worse: validateEmbeddingService checks a flag set at construction from env config (embedding-service.ts:308, :432) — "was this configured," not "can this serve a request" — so it passes while TEI is still pulling its model.

This is the root of "flaky" rather than "broken." Retries and resync intervals are compensation for a Ready signal that doesn't mean ready. We're adding a /readyz that actually checks the subsystem — the diagnostics already exist in version.ts:388-445, just wired to nothing.

What we've done and where you come in

Two PRDs, one per repo:

• vfarcic/dot-ai#714 — raised list ceiling, identity-only projection, /readyz, cheaper counting, and a published, test-enforced response contract so nobody hand-writes this shape again. That last one is the fix for the class of bug; the rest fix instances.
• vfarcic/dot-ai-controller#55 — your three fixes plus the completeness guard, progress polling, and replacing the fixtures that validated the bug. It also marks PRD PRD: Consolidate Documentation Testing Issues and Recommendations Format #34's API reference superseded, since it's currently the most discoverable description of this API and will mislead the next reader.

They're sequenced so neither blocks the other: with the truncation guard, the controller degrades safely against a server that still caps at 100.

Both of these are yours if you want them. You found this, you already have the patches, and it'd be good to have your name on the fix. I tried to assign both issues to you directly, but GitHub won't let me assign someone without repo access — happy to sort that out if you'd like them formally assigned, just say so.

To be clear though, this is an offer and not a handoff. If you'd rather I took either or both, just say so and I'll pick them up, no explanation needed. Same if you want the controller one but not the MCP-server one, or you'd rather do the parsing fix and leave the resync to me. You've already contributed a lot here, and finding the bug doesn't obligate you to fix it.

Two asks if you do take the controller work:

1. The completeness guard ships with the matching fix, in the same commit range — for the deletion reason above.
2. Fix 3 before Fix 1, per question 1.

Either way, the analysis gets credited in the PRD and changelog. And if you can share what your fork's patches settled on for the resync default and the diff guard, we'll fold that in regardless of who writes the code.

Thanks again — this was a genuinely hard one to spot from the outside, and the fact that every status surface lied made it harder.

[FlorianGerdes]

Thank you for the response and further analysis. Happy to take on the work. Feel free to add me and assign the issues.

[vfarcic]

Apologies for the slow reply — you offered to take this on and then waited on me for the one thing only I could do. Thanks for your patience.

Both are yours. I've sent you collaborator invitations on both repos:

• vfarcic/dot-ai
• vfarcic/dot-ai-controller

GitHub won't let me assign the issues while the invitations are pending — I tried, and it accepted the request and then silently kept the assignee list empty. So accept whenever suits you and I'll set both, or just say the word here and I'll do it the moment they're through.

• vfarcic/dot-ai#714 — server side
• vfarcic/dot-ai-controller#55 — controller side

Once you're in, you can push branches straight to the repos rather than working from a fork. main on dot-ai still needs a PR with one approval and squash-merges, so nothing changes about the review flow — ping me and I'll review.

The two asks from my last comment still stand, both on the controller side: the completeness guard ships in the same commit range as the matching fix, and Fix 3 lands before Fix 1.

Three things worth settling before you start on #714, since they're still open in the PRD and they shape the API surface rather than just the implementation:

1. What form the published contract takes. A JSON fixture captured from a real response is the least ceremony and the easiest for the controller's Go tests to unmarshal against. An OpenAPI schema is more discoverable and fits our existing generator, but manageOrgData is one endpoint with a union of result shapes across operations, so the schema would likely be loose enough to have let PRD PRD: Consolidate Documentation Testing Issues and Recommendations Format #34's error through anyway. I lean fixture — the failure mode we're preventing is specifically "someone hand-wrote the JSON," and only a captured response prevents that. Your call if you'd rather do it differently.
2. Whether the identity-only projection is a new operation or a parameter on list. A distinct listIds is blunt and self-documenting; a fields parameter generalizes but invites bikeshedding about which fields. The controller is the only consumer we know wants it, which argues for blunt — though dot-ai-ui also lists capabilities and might want the light payload for its table view. I'll ask on the UI side and report back, so don't block on it.
3. /readyz vs /ready. /readyz matches Kubernetes convention and the existing /healthz in the same process; /ready matches what agentic-tools already serves on its own port. Minor, but it becomes a chart contract once it ships, so worth being deliberate.

PRD #714 is still marked Draft for exactly those reasons — treat it as a spec you're free to amend rather than a finished one, and push back on anything that reads wrong from where you're sitting.

Also, unrelated but overdue: your progress-notifications PR shipped in v2.1.0 today, so that's two of yours in. And the standing offer from last time still applies — if you can share what your fork settled on for the resync default and the diff guard, I'd like to fold that in regardless of who ends up writing which piece.