feat: authenticated MCP support — built-in MCP server + per-user SSO tokens for external MCP servers

[hassan11196]

Summary

This PR adds authenticated MCP support to A2rchi in three complementary directions:

1. Built-in MCP server (services.mcp_server)

A2rchi can expose itself as an MCP server over SSE, so MCP clients (Claude Desktop, Claude Code, VS Code, Cursor) can connect directly to a deployment:

services:
  mcp_server:
    enabled: true
    url: "https://chat.example.org"

• /mcp/sse endpoint exposing 12 read-only tools: archi_query (full RAG/chat pipeline round-trip with conversation continuity), document discovery, metadata search, content grep, chunk inspection, corpus stats, deployment info, agent-spec inspection, and health check.
• Standards-based auth for MCP clients when SSO auth is enabled: OAuth2 authorization-code flow with PKCE, RFC 8414 discovery (/.well-known/oauth-authorization-server), RFC 7591 dynamic client registration, RFC 8707 protected-resource metadata, and bearer tokens backed by the mcp_tokens table.
• /mcp/auth SSO-gated page for generating/rotating tokens manually, with setup tabs for Claude Desktop / Claude Code / VS Code.
• Opt-in: with enabled: false (default) no /mcp/* routes are registered. With chat-app auth disabled, the endpoint is open (matches the rest of the app's auth model).

2. Per-user tokens for external MCP servers (mcp_servers.<name>.sso_auth)

Agent MCP tool connections can now authenticate to external MCP servers on behalf of the logged-in user:

mcp_servers:
  my_protected_server:
    transport: streamable_http
    url: https://mcp.example.org/mcp
    sso_auth: true

• After SSO login, the chat app brokers an OAuth2 authorization-code + PKCE flow against the server (/mcp/authorize, /mcp/callback), with dynamic client registration and token refresh handled by MCPOAuthService.
• Tokens are stored per user per server, encrypted at rest (pgp_sym_encrypt) in mcp_oauth_tokens.
• At query time, the agent injects a fresh Bearer token for the requesting user; sso_auth servers with no valid token are skipped gracefully (the agent still runs with the remaining tools).
• SSOTokenService persists chat-app SSO access/refresh tokens (encrypted) so MCP auth outlives short-lived Flask sessions.
• CERN CA bundle support (ssl_cert_host/ssl_cert_file config + compose mount, plus an httpx client factory) for TLS verification against internal servers.
• Guard for OpenAI's 128-tool request limit: static tools are preserved and only the MCP portion is truncated, with a warning.

3. Service-account tokens for headless interfaces (mcp_servers.<name>.service_auth)

Some interfaces have no user session and can never complete a per-user OAuth flow — the Mattermost bot, benchmarking, cron/pipeline jobs, and the startup tool registry. service_auth attaches a deployment-level static Bearer token (read from secrets) for exactly those contexts:

mcp_servers:
  unified:                       # per-user, for interactive chat
    transport: streamable_http
    url: https://remote/mcp/sse
    sso_auth: true
  unified_svc:                   # service account, for headless use
    transport: streamable_http
    url: https://remote/mcp/sse
    service_auth: true
    token_secret: UNIFIED_SVC_MCP_TOKEN

• The token is read from token_secret (default <NAME>_MCP_TOKEN); a server whose secret is unset is skipped gracefully.
• service_auth servers are deliberately skipped on user-scoped requests: interactive chat always goes through the per-user sso_auth path, so actions on the remote stay attributed to the real user, and a per-user + service twin of the same server never collide on tool names.
• sso_auth and service_auth are mutually exclusive per entry (both set → sso_auth wins, with a warning).
• The two-entry pattern (per-user unified + service unified_svc) keeps the privilege boundary visible in config.

Note: in a deployment with allow_anonymous: true, anonymous web users share the headless agent and would therefore see service_auth tools — keep anonymous access off when configuring service servers.

MCP Servers panel (Settings → MCP Servers)

A settings panel surfaces every configured external MCP server for the current user:

• Per-server connection state — active (with its tool list), needs authorization (one-click Authorize link into the OAuth flow), headless-only (service_auth on a user request), no service token, or failed — backed by a new auth-gated GET /api/mcp/status.
• A summary bar (server / active / tool / needs-auth counts) and a Refresh button.
• Per-tool read/write badges: /api/mcp/status flags tools whose remote description carries the [WRITE OPERATION] marker, with an in-list search + "write only" filter so large tool surfaces stay navigable.
• A notice when the combined tool count exceeds the 128-tool per-request limit, since beyond that the agent silently drops tools from a request.

The per-server auth gating in mcp.py is factored into a shared _prepare_server_configs, so the panel reports exactly what an agent build would do for the session.

Screenshots

Before authorization — each server's state, with a one-click Authorize for the per-user server and headless-only service servers clearly marked:

After authorizing the per-user server — the tool list with per-tool read/write badges, an in-list filter + "write only" toggle, and the >128-tool truncation notice:

Dark theme:

Schema

New tables: sso_tokens, mcp_oauth_clients, mcp_oauth_tokens (client side), mcp_tokens, mcp_auth_codes, mcp_registered_clients (server side). Defined in init.sql for fresh deployments and mirrored in ConfigService so existing deployments get them on upgrade without a manual migration. (service_auth adds no schema — it reuses the remote's mcp_tokens and a deployment secret.)

Testing

• tests/unit/test_mcp_sse_tools.py (9 tests) covering the MCP SSE tool layer.
• Live smoke test on a host-mode deployment:
  • MCP protocol over SSE: initialize → tools/list (12 tools) → tools/call for health/stats/deployment-info → full archi_query LLM round-trip returning the expected answer plus a conversation id.
  • OAuth: RFC 8414 discovery, RFC 8707 protected-resource metadata, and RFC 7591 dynamic client registration all return correct responses.
  • Client side: an sso_auth: true server with no stored token is skipped gracefully at agent init; an unreachable external server degrades gracefully without breaking the agent.
  • End-to-end per-user flow: SSO login → /mcp/authorize → remote OAuth → /mcp/callback stores an encrypted per-user token → the next authenticated message builds a request-scoped agent that loads the remote's tools.
  • service_auth: a loopback service server (the deployment's own /mcp/sse, token seeded into mcp_tokens) loads its tools through the real client path on the headless/anonymous build; the same server is correctly skipped on a user-scoped build; a server whose token secret is unset is skipped gracefully.
  • MCP Servers panel: /api/mcp/status returns correct per-server state and tool lists (including write flags) for both the headless and user-scoped views.
  • Upgrade path: on a pre-existing Postgres volume, ConfigService created all six new tables at startup.
• Unit suite: no new failures vs main (the same pre-existing environment-dependent failures occur on both).

Security review (self-review, addressed in-branch)

The branch went through an adversarial security review before this PR was opened; the following were found and fixed here:

• oauth_authorize now validates client_id + redirect_uri against the registered client (RFC 6749 §4.1.2.1). Previously an attacker-crafted authorize link could exfiltrate a victim's authorization code to an arbitrary redirect URI (PKCE does not prevent this when the attacker supplies the code_challenge).
• MCP bearer auth is now required whenever chat-app auth is enabled — previously a basic-auth-only deployment exposed the full MCP endpoint unauthenticated. The OAuth endpoint surface is registered only when bearer auth is enforced.
• Broker tokens are keyed by the SSO subject id (the users.id FK target and the id chat requests pass), not preferred_username — on IdPs where sub != preferred_username the previous keying silently violated the FK and lookups never matched.
• Token-store failures now surface to the user, and the post-login authorize chain keeps a per-session attempted set — a failing store (e.g. missing BYOK_ENCRYPTION_KEY) can no longer trap users in an infinite login→authorize→callback redirect loop.
• Agents built with per-user MCP Bearer tokens are request-scoped (thread-local) instead of being published on the shared pipeline, closing a cross-user credential-bleed race under concurrent requests.
• Also: authorization codes are consumed even when PKCE/redirect validation fails; active MCP session ids are no longer logged; the OpenAI 128-tool truncation preserves extra tools correctly; next redirect targets are restricted to relative paths; a pending sso_next redirect survives the MCP authorize chain; silent token refresh no longer extends the hard session window; stored chat-app SSO tokens are invalidated on logout; falling back from BYOK_ENCRYPTION_KEY to another secret logs a prominent warning.

Known limitations (candidates for follow-up)

• No per-server tool filtering: a remote exposing hundreds of tools overruns the 128-tool request limit, so the agent silently drops some — and which survive is list order, not relevance. A per-server allow/deny list is the natural follow-up; the MCP Servers panel now surfaces the overflow but does not yet let you curate it.
• No write-tool gating: sso_auth/service_auth tokens can carry write privileges on the remote (the panel flags [WRITE OPERATION] tools), and the agent can invoke them from a chat prompt. A confirmation gate for write tools, and scoping service accounts to read-only remote roles, are recommended follow-ups.
• Shared-pipeline concurrency: provider/model overrides still mutate shared pipeline state without a lock (pre-existing pattern, also reachable from web chat). Concurrent requests with different model overrides can interleave. The per-user credential aspect is fixed (request-scoped agents); the model/config aspect is not.
• Per-message rebuild cost: with sso_auth servers configured, each authenticated message re-initializes MCP connections (sequential get_tools per server) and recompiles the agent. A short-TTL per-user tool cache would remove most of this.
• No refresh-token-rotation locking: two concurrent messages can race the silent refresh; with rotating refresh tokens the loser can invalidate the grant.
• Broker spec gaps: no resource (RFC 8707) or scope parameter in the brokered authorize/token exchanges; discovery ignores URL paths for path-mounted servers. Remote-issued mcp_tokens are long-lived and report no expires_in, so the broker assumes a 1-hour lifetime and re-auths on expiry.
• sso_tokens currently has no consumer (stored for upcoming per-user Bearer use); rows are encrypted at rest and invalidated on logout.