For the owner of an installation and for a security reviewer. The meta chart ships Agent Substrate — the runtime of the kagent line on api.kagent.dev — as two components next to kagent (reference: Agent Substrate). This page is the honest account of what that puts on the cluster and its nodes, which of it steps outside the restricted Pod Security Standard and why, what compensates, and which risks are open upstream.
Every agent of the platform runs as a Substrate actor: a gVisor sandbox inside a long-running worker pod of a WorkerPool, resumed from a snapshot when a request arrives and suspended back to the object store when idle. A control plane in ate-system — ate-api-server (the actor and worker registry, backed by Postgres), ate-controller (turns WorkerPools into worker Deployments), the atenet router (a turn's ingress: it asks ate-api which worker hosts the actor and tunnels the request to it), the atenet egress gateway (every connection an actor opens leaves through it) and dns — and a per-node agent, atelet, which restores and snapshots sandboxes, pulls the actors' images and the gVisor runtime, and registers the CSI and device plugins the workers use. Identities between all of these are pod certificates (PodCertificateRequest, a Kubernetes 1.35 beta API) signed by a podcertificate-controller from CA pools the chart bootstraps; actors authenticate to ate-api with the cluster's ServiceAccount tokens.
| Change | Where | Why |
|---|---|---|
Kubernetes 1.36 with three feature gates (PodCertificateRequest, ClusterTrustBundle, ClusterTrustBundleProjection) on kube-apiserver, kube-controller-manager and every kubelet |
the cluster (giantswarm/cluster#1005, or the cluster chart's internal.advancedConfiguration.*.featureGates until then) |
the pod identities of every Substrate component and worker are PodCertificateRequests the kubelet issues and projects; the trust anchors are ClusterTrustBundles. Enabling the gates rolls the control plane and every node. |
A root DaemonSet (atelet, not privileged: all capabilities dropped but DAC_OVERRIDE) on every schedulable node, with hostPorts 8085 (its gRPC API, mTLS) and 9090 (metrics) and hostPath mounts of /var/lib/ateom-gvisor (the sandboxes' state, shared with the workers), /var/lib/kubelet/plugins, /var/lib/kubelet/device-plugins (the CSI and device plugins it registers) and /dev (read-only) |
ate-system |
it runs gVisor on the node's behalf: creates and restores sandboxes, mounts their filesystems, wires their network into the worker pod. |
Worker pods as root with the capability set an unprivileged gVisor sandbox needs (NET_ADMIN, SYS_ADMIN, SYS_CHROOT, SYS_PTRACE, SETUID, SETGID, SETPCAP, DAC_OVERRIDE, FOWNER, CHOWN, MKNOD, NET_RAW, SETFCAP), AppArmor and seccomp Unconfined, a hostPath of /var/lib/ateom-gvisor with HostToContainer propagation — not privileged |
the kagent namespace (every WorkerPool) | runsc installs its own seccomp filters and needs the capabilities to build the sandbox; the propagation shares the node agent's mounts into the worker. The sandbox, not the pod, is the isolation boundary for the agent's code. |
A cluster-scoped SandboxConfig naming the gVisor release asset atelet downloads (gs://gvisor/releases/…, verified by sha256) and a ValidatingAdmissionPolicy that holds SandboxConfigs to a valid shape |
cluster scope | the runtime binary is not an image; a proxied installation mirrors the asset and points spec.assets at the mirror. |
Three CRDs (workerpools, sandboxconfigs, csidriverconfigs.ate.dev, the substrate-crds chart), ClusterRoles for ate-controller (pods, Deployments, WorkerPools), atelet (pods, ClusterTrustBundles, SandboxConfigs, read-only) and ate-api-server, and the podcertificate-controller's signer role |
cluster scope | |
Two namespaces, ate-system and podcertificate-controller-system, holding the CA and JWT pools (Secrets) the platform bootstraps once and keeps |
the pools are the roots of every identity Substrate issues; access to those two namespaces' Secrets is access to those roots. | |
Egress from the nodes to storage.googleapis.com (the gVisor asset), the image registries of the actors' images, and the snapshot store |
atelet |
A Giant Swarm cluster enforces the restricted standard through Kyverno. The connectivity chart ships one PolicyException per Substrate workload, each naming exactly the rules that workload violates — make verify-kyverno computes the violations from the rendered pod specs and fails the build when an exception is broader or narrower:
| Workload | Rules excepted |
|---|---|
atelet (DaemonSet, ate-system) |
host-ports-none, host-path, restricted-volumes, adding-capabilities-strict, run-as-non-root, run-as-non-root-user, privilege-escalation, check-seccomp-strict |
the worker pods (every WorkerPool, label ate.dev/worker-pool, the kagent namespace) |
host-path, restricted-volumes, adding-capabilities, adding-capabilities-strict, run-as-non-root, run-as-non-root-user, privilege-escalation, check-seccomp, check-seccomp-strict, app-armor |
the control plane (ate-api-server, ate-controller, atenet-router, atenet-egress, dns) |
require-drop-all, run-as-non-root, privilege-escalation, check-seccomp-strict — the Deployments declare no securityContext; the images are distroless and run as non-root users, the fields are what the standard checks |
podcertificate-controller |
run-as-non-root, check-seccomp-strict |
the credential provider (k8s-credential-provider, ate-system) |
check-seccomp-strict — non-root, drops ALL, no privilege escalation, read-only root, but no seccompProfile |
Nothing else is excepted: no app: kagent selector remains (the v1alpha2 agent Deployments' exception is gone with them), the exceptions are scoped to the two Substrate namespaces (the control plane matched by workload name — its Deployments carry no labels of their own) and the WorkerPool label in the kagent namespace, and the hook Jobs of the connectivity release run under the restricted profile themselves.
With tenants.organizations.enabled (agent-manager's tenants, #1036) a second exception, substrate-workers-tenants, matches the same WorkerPool label in every namespace labelled giantswarm.io/organization, by a namespaceSelector (Kyverno 1.15.2 or later), and the router, the egress gateway and the dns admit those workers as they admit the kagent namespace's. An Organization's admins may write pods in its namespace, so there the exception applies only when Substrate's controllers write the object: ate-controller the Deployment (and the busy workers' pods), the deployment controller the ReplicaSet (and the Deployment), the ReplicaSet controller the Pods. A Deployment, ReplicaSet or Pod with the label that a person writes there is judged like any other. Kyverno's autogen rules judge a ReplicaSet at its admission, which keeps a person's ReplicaSet from reaching the ReplicaSet controller. A background scan has no writer, so this exception sets background: false (Kyverno refuses subjects otherwise) and the scans report these workers as violations; substrate-workers keeps the kagent namespace's workers excepted in them. The controllers are named by their ServiceAccounts, which assumes kube-controller-manager runs with --use-service-account-credentials (kubeadm's default); without it the workers are refused.
With the same value, Substrate's credential provider reads the Secrets of every namespace labelled giantswarm.io/organization for two atespaces (#1041): the atespace of the namespace's own name, and no other tenant's; and ate-golden, the one atespace where every template's golden actor boots, which reads all of them. That second grant crosses tenants: the provider sees only the atespace and the template's UID, so it cannot tell which tenant a golden actor serves. The guard is kagent's translation of a template: it resolves the template's model, MCP servers, skills and shared templates in the template's own namespace, so a golden actor's egress policy names Secrets of that namespace only.
ate-controller builds the worker Deployment from the WorkerPool, and in such a namespace an Organization's admins hold cluster-admin, as does rbac-operator's automation ServiceAccount that the members bound to agent-platform:editor reach. RBAC cannot take that away, so two ValidatingAdmissionPolicies of the connectivity release (templates/substrate/tenant-workers-admission.yaml, rendered with the value and Agent Substrate on) hold the writes in every namespace labelled giantswarm.io/organization except the kagent namespace, for every requester:
agent-platform-tenant-workers: only agent-manager's ServiceAccount (the tenant reconciler) creates, updates or deletes a WorkerPool; the namespace and garbage collectors may delete one. Only the writers the exception names create or update a Deployment, ReplicaSet or Pod that carriesate.dev/worker-poolon itself or on its pod template, before or after the write; the garbage collector may update one, because an orphaning or foreground delete edits owner references and finalizers and stays stuck otherwise. No one adds an ephemeral container to such a pod, and no one execs into or attaches to a pod named after the platform WorkerPool (<kagent.substrateWorkerPool.name>-*, the workers: a CONNECT request names the pod but carries none of its labels, so a tenant's own pod with such a name is held too; the render refuses a pool name longer than 57 characters, which the API server would cut from the pods' names). This holds for Giant Swarm staff too: a tenant's worker is debugged through its logs or a node debug pod (kubectl debug node/<node>), never from inside. Deletes, the status and scale subresources and port-forward stay with RBAC.agent-platform-tenant-workerpool-copy: a WorkerPool there is a copy of the platform WorkerPool (kagent.substrateWorkerPool.namein the kagent namespace, the policy's parameter): the same name and spec, onlyreplicasmay differ; without the platform WorkerPool every WorkerPool write there is refused. The reconciler copies that pool and nothing else, so this changes nothing for it, and it keeps agent-manager's ServiceAccount, which serves a network API, from choosing what runs on a node.
A tenant's workers therefore run what the platform WorkerPool held when agent-manager last wrote the tenant's copy (the policy judges a WorkerPool when it is written, not when the platform pool changes), and only Substrate's controllers make them. Kyverno and these policies are separate validating steps of the same request: each refuses on its own, and mutating webhooks run before both. ate-controller's other writes (the WorkerPool's status, the worker pods' busy label) and the kube-controller-manager's (the Deployment's revision, the ReplicaSet, the Pods) pass.
- The sandbox is the boundary. An agent's code runs under gVisor (
runsc), a user-space kernel; the worker pod's capabilities serve the sandbox's construction, not the agent. The worker pod is a long-lived shell that hosts up to--max-actorsactors (default 1000) and is replaced by the controller when the pool rolls; it carries adefaultServiceAccount token that nothing inside the sandbox can reach without escaping it. - Every hop is authenticated. Pod certificates (mTLS) between
atelet,ate-api-server, the router and the egress gateway; actors authenticate to ate-api with the cluster's ServiceAccount tokens for the audienceapi.ate-system.svc, verified against the apiserver's issuer; the router authorizes a request against the actor assigned to the worker before it tunnels it; the egress gateway resolves the connecting actor and applies the egress policy in the request path. - Network policies in both flavours (
templates/substrate/netpol.yaml): the control plane accepts only its peers; the worker pods reach only the egress gateway, the dns and the cluster DNS — an actor's outbound connections all leave throughatenet-egress, where the actors' allow-list lives (muster, the kagent controller, the LLM provider or the agentgateway LLM listener, DNS); the kagent controller admits the gateway on its API port and nothing else of Substrate's. The router admits a turn on 8080 only from the kagent controller (app.kubernetes.io/name: kagent,app.kubernetes.io/component: controller) in the namespace the kagent chart renders it into,kagent.namespaceOverride(the meta chart'skagent); any other pod, a debugging pod in the same namespace included, is dropped, while akubectl port-forwardto the router enters the pod's network namespace and is not subject to the policy, so it proves the router, not the path. Rendered on every installation; enforced where Cilium enforces (agentlab does not). - Namespace scoping. Substrate's control plane and its pools live in two namespaces of their own; the WorkerPool and the workers in the kagent namespace with the agents' other objects; the hook that bootstraps the pools holds its rights for the seconds it runs (a ClusterRole on secrets, configmaps and namespaces, created and removed with the hook).
- Roots that are minted once. The CA and JWT pools are created when missing and never rotated by the chart — a rotation is an operational procedure of Substrate that re-issues every dependent identity, not something an upgrade may do by accident.
- One architecture per pool. The WorkerPool is pinned to the installation's CPU architecture: a mixed pool wedges the actors whose snapshot was taken on the other.
- Render-time refusals. The chart refuses kagent without Substrate, Substrate without its CRDs or a database, kagent without a snapshot store, and — where the render is live — a cluster that does not serve
certificates.k8s.io/v1beta1/PodCertificateRequest.
An actor's outbound traffic is whatever its egress policy declares, on any TCP port, plus the listed in-cluster destinations (muster, the kagent controller, the agentgateway LLM listener, the OTLP gateway, the cluster entity on 443 and 10443). substrate-atenet-egress opens world on every TCP port: the egress gateway, which resolves the connecting actor and applies its egress policy in the request path, is the control on where an actor connects, and the cluster policy does not second-guess it with a port list of its own.
| Request | What the actor sees |
|---|---|
An origin the actor's egress policy declares, on any port (https://host:8443, http://host) |
the response |
| A destination the actor's egress policy does not declare, on any port | 403 at once, naming the policy denial |
The instance metadata services (169.254.0.0/16, fd00:ec2::254) and Azure's WireServer (168.63.129.16) are denied to the egress gateway (egressDeny), whatever an egress policy declares: they hand out the node's cloud identity. A host model server is opened by its own rule in the chart, from the address the ModelConfigs carry, since it may sit on a node of the cluster, which the world entity does not include.
Substrate is an early project; its own threat model states that it "has little to no security hardening at this time". What matters for an installation:
- Worker-pod host escape — an open Critical in the threat model: an actor that escapes gVisor lands in a root pod with
SYS_ADMINand a host mount. The mitigating invariant upstream names (privileged operations moved out of the worker pod, or the sandbox's construction done by the node agent) is not implemented at the pin; the network policies and the pod's placement (a WorkerPool node pool,kagent.substrateWorkerPool.template) are what an installation can add today. - No authorization in ate-api. Every authenticated caller of ate-api has full control of every atespace, actor, template and worker; the kagent controller is the only intended caller, the network policies keep it so, and the JWT provider admits only the cluster's ServiceAccount tokens for the ate-api audience.
- Control plane next to the workers. The threat model recommends running the control plane and the egress gateway on nodes the sandboxes do not share;
substrate.atelet.nodeSelectorand the WorkerPool'stemplate.nodeSelectormake that a node-pool decision of the installation. - The gVisor runtime as a downloaded asset.
ateletfetchesrunscfromstorage.googleapis.comat prewarm, verified by the sha256 theSandboxConfignames; the asset is not an image and not scanned by the registry's scanner. - Egress for an actor while it resumes — the carried patch of the Substrate line that lets a harness fetch its skills before it is ready; complete with the egress dataplane the pinned release runs (the agentgateway line's build the Substrate release stamps as
images.agentgateway, which authorizes a resuming actor at CONNECT time as a frontend policy; the line's ledger, giantswarm/giantswarm#37742 rows 8 and 13). - Everything the platform carries against upstream, with its upstream exit: giantswarm/giantswarm#37742 (the Substrate rows) and the line's
FORK.md.
- Kubernetes 1.36 and the three gates on all three components, before Substrate is enabled.
- The snapshot store: an S3 bucket and an IRSA role for
ateletandate-api-server(CAPA —kagent.harness.snapshotStore.crossplanerenders both through Crossplane and derives the location; CAPZ —provider: capzrenders the storage account, the container, the Workload Identity and the s3proxy façade Substrate talks S3 to, admitted only from atelet and ate-api-server), or an S3-compatible store with its credentials in a Secret; otherwisekagent.harness.snapshotLocationnames it. - Node placement, when the installation dedicates a node pool to the sandboxes:
substrate.atelet.nodeSelector/tolerationsandkagent.substrateWorkerPool.template. - Nothing else: the bootstrap, the database on the platform's CNPG Cluster, the exceptions and the policies come with the chart.