Add untrusted mode: MCP servers that never possess credentials

[JAORMX]

Summary

You want to run a third-party MCP server you don't trust. Today, that means handing it real credentials — in its environment, in files, or in request headers — and hoping. This PR adds untrusted mode: an MCP server that never possesses a credential at all. Not a long-lived one, not a short-lived one, not even for a millisecond.

The experience, by persona:

The end user notices almost nothing. You ask your agent to use the GitHub MCP server. Behind the scenes, ToolHive spins up a backend pod just for you, and your GitHub token is injected into your API calls at the cluster's egress boundary — the server running your request never sees it. First time using a provider? The tool result comes back with a "connect your account" link instead of a cryptic failure; you click, consent, retry, done.

The cluster operator gets a declarative, admission-guarded API. Two blocks on the MCPServer and the platform handles the rest:

spec:
  untrusted: true
  groupRef: github-group        # CEL-enforced: must sit behind a VirtualMCPServer
  egressPolicy:
    providers:
    - provider: github
      allowedHosts: [api.github.com]
      allowedMethods: [GET, POST]   # empty = read-only by default
      credentialEnvName: GITHUB_TOKEN

Invalid configurations (secrets on the backend, custom pod templates, missing group) are rejected at apply time, not discovered at 2am. The operator provisions and rotates the TLS CA, generates NetworkPolicies (default-deny egress: loopback, cluster DNS, and the allowlisted destinations only), and exposes four alert-worthy signals: quota-denied admissions, pod-fan-out approaching caps, credential-echo suppression events, and consent-flow breakage.

The attacker running inside the compromised MCP server finds a pod with a fake token (GITHUB_TOKEN=thv-untrusted-sentinel:github), no neighbors (one user per pod), no network (everything but the sidecar is denied), and no way to name another user's session — the token-selection key never crosses their process in any form. Calls to non-allowlisted hosts are denied before any credential is loaded. Responses that echo the injected credential back are replaced with a generic 502. Every injected and denied call lands in a structured audit log.

The MCP server author changes nothing: the token env var contains a sentinel, make the API call anyway, the platform injects the real credential.

What this defeats: credential theft, replay, cross-user use, and post-hoc use. What it does not defeat (documented loudly in the ADR and user docs): authority abuse within the granted scopes for the current call, data exfiltration through MCP responses, and encoding-transform evasion of the byte-exact response scanner. Least-privilege OAuth scopes, method/path ACLs, and the audit trail are the mitigations.

Type of change

• New feature

Test plan

• Unit tests (task test) — full suite with -race green, including new packages (pkg/egressbroker, pkg/vmcp/session/untrusted)
• Linting (task lint-fix) — clean
• envtest integration suites (cmd/thv-operator/test-integration/mcp-server-untrusted, mcp-external-auth; pkg/vmcp/session/untrusted integration-tagged suite) — green
• Manual testing (describe below)

Manual: matlatl docs link-graph check clean. Not yet run: the chainsaw e2e scenarios (test/e2e/chainsaw/operator/untrusted-egress, validation/mcpserver-untrusted) and a live-Envoy end-to-end through the TLS-bump path — both need a Kind cluster. These are wired into CI config but are the known verification gap before merge; flagged for reviewers.

API Compatibility

• This PR does not break the v1beta1 API

All CRD changes are additive-with-defaults: MCPServer.spec.untrusted (default false), spec.egressPolicy (optional), AuthServerStorageConfig.tokenEncryption (optional). CEL rules gate only untrusted=true workloads; trusted workloads are byte-identical in behavior.

Changes

Area	Change
cmd/thv-operator/api/v1beta1	untrusted + egressPolicy fields with CEL R1–R6; tokenEncryption on auth-server storage
cmd/thv-operator/controllers	Sentinel injection, untrusted resources (generation-named CA, policy ConfigMap, NetworkPolicy), terminal/transient error discipline
pkg/vmcp/session/untrusted	Single-tenant pod lifecycle: template-clone pod creation, DoS admission (Redis ledgers), reaper GC, address resolution
pkg/egressbroker + cmd/thv-egressbroker	Envoy ext_authz credential injector (destination binding before load), SDS TLS-bump, ext_proc response scanner, audit events, OTel metrics
pkg/authserver/storage, pkg/authserver/tokenenc	Read-side binding validation (+Strict mode), AES-GCM envelope encryption of upstream tokens at rest
pkg/auth/upstreamswap, pkg/vmcp/auth	Consent-on-demand: actionable UPSTREAM_CONSENT_REQUIRED errors with authorize URL in both subsystems
docs/arch/15-untrusted-mode.md, docs/operator/untrusted-mode.md, docs/arch/adr/0001	Architecture, operator guide, and ADR with the security boundary stated explicitly

Does this introduce a user-facing change?

Yes — a new opt-in mode for running untrusted MCP servers on Kubernetes. Cluster operators mark an MCPServer untrusted: true with an egressPolicy; end users keep their upstream OAuth tokens entirely out of the MCP server's reach, with a guided consent flow when a provider isn't connected yet. See docs/operator/untrusted-mode.md.

Special notes for reviewers

• Scale: ~26k lines across 7 commits (one per wave; the commits tell the story in order: security foundations → CRD+lifecycle+broker → consent → production readiness → response scanning → docs → review hardening). Commit-by-commit review is likely easier than the full diff.
• The security invariants to attack: (1) no credential material in the backend pod — env, files, headers; (2) destination binding evaluated before credential load (pkg/egressbroker/injector.go); (3) token selection never influenced by anything the server emits (single-tenant pods + pod-annotation identity); (4) fail-closed everywhere except the documented scanner fail-open default.
• Review coverage already applied: each wave passed adversarial panels (security, operator, QA/test, architecture, spec, standards) plus a final full-branch duplication + library-reuse pass. Their findings are fixed in-place; notable catches included an Envoy bootstrap that couldn't actually TLS-bump, a quota that never enforced, and correlation metadata that would have left the scanner inert.
• Known residuals (documented in ADR-0001 §"security boundary"): authority abuse within scope, data exfil via MCP responses, encoding-transform scanner evasion, and the pre-existing TestTokenMap_ConcurrencyFlood flake (racy assertion, fails on baseline too — follow-up candidate).
• Follow-ups not in this PR: Kind/chainsaw e2e execution, trusted-path NetworkPolicy generation from permissionProfile (now cheap with this machinery), shared egress-policy CRD for multi-workload reuse.