Inventory of every hack/workaround in the lab's scripts, what it papers over, and what happened to it. One commit per resolved item. Statuses:
2026-08-27, Go port: the lab was rewritten as a single Go binary (
agentlab) with form-driven configuration; scripts/ and manifests/ are gone. Every fix below carries over — the checksum stamping (H2/H3), the digest gate (H9), the exact-tag image check (H1), key permissions (H10, now 0600 at creation) and the post-render patches (U1-U3, nowagentlab post-render) all live ininternal/lab/. Two NEW items surfaced during the port: H11, H12.2026-09-08, the agent-platform meta chart in the lab shape: the lab installs
giantswarm/agent-platform(bundled Flux engine on, self-management off) instead of the git-vendored agent-platform-standalone umbrella.fluxUp, the vendored chart (.vendor/), the post-renderer binary and its generated Helm plugin (state/helm-plugins/) are gone; the post-render patches (U2, U9, U13) are per-componentpostRenderersvalues the chart forwards to the component HelmReleases (postrenderers.go), the image preload resolves the component charts from the rendered OCIRepositories (fluxreleases.go), and mcp-prometheus is a lab-rendered HelmRelease through the same engine. H12, H13, U3 and U4 are retired with the mechanisms they patched.
- FIXED — replaced with a proper solution (commit referenced).
- BLOCKED UPSTREAM — cannot be fixed in this repo; the exact upstream change that unblocks it is named.
- ACCEPTED — deliberate lab trade-off, not worth "fixing"; reasoning given.
docker exec … crictl images | grep -q giantswarm/backstage matches any
backstage image. Bumping IMAGE to a new tag would silently keep running the
old one already loaded into the node.
Fix: exact repo:tag match against crictl images -o json repoTags.
The dex-ca secret was created after kubectl apply -f manifests/backstage.yaml,
so on first boot the pod sat in ContainerCreating waiting for a volume that did
not exist yet; an unconditional rollout restart then bounced the (2.4 GB,
Rosetta-emulated) pod on every re-run to compensate for possible cert changes.
Fix: namespace + secret are applied before the Deployment, and a
checksum/lab-inputs annotation (sha256 over the manifest + certs/ca.crt)
is stamped into the pod template — same pattern dex.yaml already uses — so
the pod rolls exactly when the config or CA changes and no-op re-runs stay
no-ops. The blanket rollout restart is gone.
The sed "s/REPLACED_BY_UP_SH/$SUM/" manifests/dex.yaml | kubectl apply logic
lived in two places (up.sh and the reload target) and would drift.
Fix: extracted scripts/apply-dex.sh (stamp + apply + rollout wait);
up.sh and make reload both call it. The checksum now also covers
certs/tls.crt, so a cert rotation rolls the pod too. Placeholder renamed to
REPLACED_AT_APPLY to match its new owner.
Same ordering smell as H2: on first boot the Dex pod waited on a secret that was applied a step later. Fix: namespace + dex-tls secret are applied before the Deployment.
H5. restart-apiserver.sh: blind sleep 5 between manifest removal and restore — FIXED, then superseded by H6
The static-pod bounce moved the manifest away, slept 5 seconds, and moved it
back. If the kubelet had not yet noticed the removal, the restore was a no-op
and the apiserver never restarted — a silent false success.
Fix: poll crictl ps inside the node until the kube-apiserver container
is actually gone before restoring the manifest, fail loudly on timeout, and
restore the manifest via an EXIT trap on every path so the cluster is never
left without an apiserver.
The whole reason for restart-apiserver.sh was the claim that the OIDC
authenticator gives up discovery after ~40 s and never retries. Verified
false on a cold boot (kind v0.31, Kubernetes 1.35, 2026-08-27): the apiserver
logs oidc.go:433 … initializing plugin: … connection refused every 10
seconds indefinitely, and the first admin token was accepted immediately once
Dex answered — zero bounces. The 40s/4-retries behavior belonged to older
Kubernetes.
Fix: restart-apiserver.sh deleted, the call removed from up.sh, the
restart-apiserver Make target removed, README gotcha rewritten. up.sh's
existing 60 s verification loop comfortably covers the ≤10 s window until the
authenticator's next retry tick.
The for … seq 1 40 loop printed nothing and fell through silently when the
MCPServer never reached Connected; the script then declared the platform up.
Fix: the loop now reports the last observed state and exits non-zero on
timeout, pointing at make platform-logs.
Response headers/bodies went to /tmp/.mcp-h / /tmp/.mcp-b — collision-prone
between concurrent runs and left behind afterwards.
Fix: mktemp + trap … EXIT cleanup.
[[ Chart.lock -nt charts ]] plus a touch charts stamp. mtimes lie (git
checkout order, copies, clock skew) and the failure mode is a stale charts/
dir silently installed.
Fix: store the sha256 of Chart.lock in charts/.lock-digest after a
successful helm dependency build and rebuild whenever it differs.
chmod 644 certs/*.key made the CA and server keys world-readable. Nothing
needs that: the kind mount is read by root in the node container regardless,
and the secrets are created from file by the invoking user.
Fix: 600 on keys, 644 on certs; re-runs also tighten keys left
world-readable by older versions of the script.
Latent in the shell lab and exposed by the Go port's user-editing flow: a
config change rolls the Dex pod by design (checksum annotation), and with
in-memory storage the new pod mints new signing keys — the apiserver then
rejects every token with failed to verify id token signature until its
JWKS cache refreshes (observed: minutes). The old verification never noticed
because make reload was only ever tested as a no-op; edit a user and reload,
and all logins broke.
Fix: Dex now uses its CRD-backed kubernetes storage (ServiceAccount +
ClusterRole for the dex.coreos.com API group). Keys persist across rolls, an
immediate post-reload login verifies, and the state still dies with the
cluster.
The agent-platform-standalone checkout lived in vendor/, which flips a Go
module into vendored-build mode: after the first agentlab platform, go build
failed with "inconsistent vendoring".
Fix: the chart was vendored into .vendor/ instead. Retired 2026-09-08:
the lab installs the released agent-platform chart from the registry (or a
local chart directory, platform.chartPath); nothing is vendored, and
agentlab platform removes a leftover .vendor/.
H13. agentlab platform failed on Helm 4 server-side-apply conflicts after a dev-image swap — RETIRED
The documented way back from a dev-image swap (kubectl set image /
kubectl patch on a platform Deployment, see the agentlab skill) was "re-run
agentlab platform, which reconciles every Deployment to the vendored chart".
Helm 4 applies server-side, so the swapped image field belongs to the
kubectl field manager afterwards and the upgrade died on the conflict
(Apply failed with 1 conflict: conflict with "kubectl-set" ... image); the
workaround was deleting the Deployment by hand and re-running.
Fix: the umbrella upgrade passed --force-conflicts: Helm took the
fields it renders back from any other manager, so a re-run really did
reconcile the lab to the chart. Retired 2026-09-08: the workloads are the
component HelmReleases' now (helm-controller applies them), and the dev-image
loop goes through the values — platform.devImages renders a kustomize image
override into the component's postRenderers, so the swap and its restore are
plain helm upgrades with no second writer and no flag. A hand kubectl patch
lasts until helm-controller's next release of that component.
U1. muster-post-render.sh patch: allowPublicClientRegistration edited into the rendered ConfigMap — FIXED upstream
The muster chart exposed the key in values.yaml, values.schema.json, README
and unit tests, but templates/configmap.yaml never rendered it, so the value
was silently dropped and Claude Code's DCR registration died with
"Registration requires authentication". Neither alternative gate can work for
a public loopback client (no token, random port, http/https stripped from
allowed schemes by mcp-oauth validation), so the key was edited into the
rendered ConfigMap by the post-renderer.
Fixed: muster 5.7.2 (giantswarm/muster#1118) renders the key; the chart
BOM carries it since the curation that pinned muster 5.7.2. The post-render
ConfigMap edit is deleted — the values entry alone is effective now.
muster must resolve https://localhost:32000/dex to the Dex NodePort from
inside the pod (the lab's one-issuer-URL trick), which needs hostNetwork; and
with hostNetwork a default rolling update deadlocks on a single node because
both pods want :8090. For the same reason the lab runs one replica: the
agent-platform chart's HA default of two muster replicas would leave the
second pod Pending on the single node and the HelmRelease never Ready. The muster chart (checked through 5.6.2) exposes no
hostNetwork, dnsPolicy or strategy values; the Backstage chart neither. The
option platform.musterPodNetwork drops the patch for muster: two replicas on
the pod network, the issuer bridged by the U13 sidecar.
Since 2026-09-08 the patch is a Kustomize strategic merge in
components.muster.postRenderers / components.backstage.postRenderers
(postrenderers.go), which the agent-platform chart forwards to the component
HelmRelease and the bundled helm-controller applies — no post-renderer binary.
Unblocks: giantswarm/muster — add deployment-level hostNetwork/
dnsPolicy/strategy values. Until then the patch is surgical so chart bumps
need no hand-copying.
agent-platform-standalone's _helpers.tpl hard-failed on empty
ingress.parentRefs in all modes — even muster-direct, where the lab
had no Gateway at all. The values carried a no-gateway-in-this-lab
placeholder to pass the guard and the rendered route was stripped.
Retired: the lab runs the chart-owned agentgateway edge
(gatewayApi.gateway.create: true), so the routes attach to a real Gateway and
nothing is stripped; the post-renderer that did it is gone (2026-09-08).
The umbrella chart had no OCI release: charts/giantswarm/agent-platform-standalone
did not exist in gsoci (verified 2026-08-27, NAME_UNKNOWN), so the lab
vendored it from git at a pinned SHA (platform.apsRef).
Retired 2026-09-08: the lab installs the released agent-platform meta
chart — helm upgrade --install … oci://gsoci.azurecr.io/charts/giantswarm/agent-platform --version <platform.chartVersion> — or a local chart directory
(platform.chartPath) for unreleased changes; the standalone umbrella retires
with the one-chart delivery story.
muster dials its MCPServers ~2s after starting; a failed first dial schedules
a retry "after 30s" but the orchestrator's backoff-expiry sweep only fires
~60s later ("Attempting to reconnect failed MCPServer … (backoff expired)" at
+60s, observed 2026-08-28 on muster 5.5.6). Overlapping the two helm installs
therefore added ~30s to boot: muster started before mcp-kubernetes was
ready and ate the fixed penalty. The workaround was ordering — a separate
mcp-kubernetes release --waited before the umbrella so muster's first dial
always succeeded.
Retired: since agent-platform-standalone#44 the umbrella bundles
mcp-kubernetes itself (components.mcp-kubernetes, on by default, registering
an MCPServer named mcp-kubernetes, no muster family), and the lab adopted it
(#34): the standalone release, its version pin and its values template are
gone. The two workloads now start concurrently inside
one release, so a muster pod that wins the race can still eat the reconnect
backoff before the MCPServer shows Connected — the boot's 120s Connected wait
absorbs it. The muster-side fix (sweep at the scheduled retry time) would
remove that residual delay.
A muster pod (5.5.6, mcp-oauth v1.3.1) can come up with a TLS trust pool that
is missing the --extra-ca-file CA: every /mcp bearer is then rejected with
invalid_token — the JWKS fetch AND the userinfo fallback both fail with
x509: certificate signed by unknown authority against Dex — while
everything the lab used to check looks healthy (rollout done, MCPServer
Connected, OIDC discovery succeeded, /.well-known/oauth-authorization-server
answers 200). Observed 2026-08-28; the same pod's OIDC discovery over the
same endpoint succeeded, so only some client constructions miss the CA.
Verified not to be config: the dex-ca secret matched certs/ca.crt and Dex
served the lab-CA-signed cert at the moment the pod failed x509. Not
deterministic — a pod restart with identical inputs validated fine, and a
forced degraded-start reproduction (valkey down at muster start) also
validated fine, so it is an in-process race in muster/mcp-oauth client
construction, not the degraded-recovery path per se.
Workaround: after the install, up now probes the real token path
(password grant -> Bearer on /mcp, ensureMusterValidatesTokens) and, on
rejection, replaces the muster pod once and re-probes.
Unblocks: giantswarm/muster — find the client construction that misses
ExtraCAFile/RootCAs (candidates: anything cloning http.DefaultTransport
before bootstrap swaps it, or an mcp-oauth client built without
opts.RootCAs) and thread the pool deterministically; then the probe can
stay but the bounce becomes dead code.
The GS Backstage build up to 0.244.x deployed an agent by scaffolding the
hidden catalog Template template:default/agent-deployment (a kube:apply
of manifests the portal composed itself); the image shipped no such entity,
so the lab embedded a verbatim copy of the upstream template into its
catalog ConfigMap (internal/lab/templates/static/, the staticFile template
func). Retired with the kagent API v2 portal (backstage 2.x): the wizard's
Deploy is agent-manager's create_agent through the muster plugin's backend
as the signed-in person — no scaffolder task, no kube:apply, no manifest
composed in the portal — and agentlab backstage-test drives exactly that
path (docs/backstage.md "The agent create flow"). The embedded copy, the
catalog location and the template func are deleted; the lab catalog carries
the users and groups only.
kagent 0.x's controller composed the runtime image for runtime: go agents
from its own version tag (IMAGE_REGISTRY/golang-adk:IMAGE_TAG), which
Giant Swarm's retagging lagged for exactly that repo, leaving every created
agent in ImagePullBackOff; healADKImages (internal/lab/adk.go) pulled
the tag or stood in the newest published older release and side-loaded it.
Retired with kagent API v2: the platform Harness pins its runtime image by
digest (the connectivity chart's kagent.harness.image, moved with the line's
re-pins), the controller composes no image from its tag, and there is no
agent pod — agents run as Substrate actors from the Harness's image. The
heal and its file are deleted; the image reaches the node through the
preload (preload.go) like every other of the platform's.
The lab host-publishes the kagent UI through a kind port mapping, which needs
a stable node-side port. ui.service.type: NodePort is a chart value, but
the upstream kagent chart's ui-service.yaml template renders no nodePort
field (verified in kagent 0.9.12 via the vendored wrapper chart 0.1.37), so
Kubernetes assigns a random one — useless to kind's fixed extraPortMappings.
A Kustomize strategic-merge patch in components.kagent.postRenderers
(postrenderers.go, since 2026-09-08; agentlab post-render before) pins the
Service's ui port entry to config.KagentUINodePort (30880); the kind config
maps that onto platform.agentsPort (default 8081) on the host.
Unblocks: kagent-dev/kagent — render ui.service.ports.nodePort when set
(the standard chart idiom). The value then moves into
agent-platform-values.yaml.tmpl and the patch is deleted.
The agentgateway edge (the chart-owned Gateway under
gatewayApi.gateway.create) is host-published through a kind port mapping,
which needs a stable node-side port. The data-plane Deployment/Service are
created by the agentgateway controller reconciling the Gateway — they are not
part of the Helm release, so neither chart values nor agentlab post-render
can pin the controller-created Service's NodePort. The lab applies its own
Service (gateway-nodeport.yaml.tmpl, NodePort config.GatewayNodePort
30443) selecting the data-plane pods by the standard
gateway.networking.k8s.io/gateway-name label the controller stamps on them;
the kind config maps 30443 onto platform.gatewayPort (default 443).
Fragile if: the controller changes its generated pod labels.
Unblocks: an agentgateway/AgentgatewayParameters knob for the generated
Service's nodePort; then this Service is deleted and the kind mapping targets
the controller's own Service.
The kagent 0.x package pinned upstream 0.9.x CRDs whose v1alpha2 Agent spec
predated the A2A-card metadata fields; the Backstage create flow composed
agent.iconUrl, the 0.x agent chart rendered it into Agent.spec.iconUrl,
and server-side apply rejected the whole HelmRelease. patchAgentCRDIconURL
(internal/lab/kagentcrd.go) added the property to the installed CRD.
Retired with kagent API v2: no agents.kagent.dev is served any more, and
the Generic agent chart 1.x renders agent.iconUrl as the
ui.giantswarm.io/icon-url annotation on the AgentTemplate (which has no
icon field) — the Dev Portal reads the annotation, agents-test asserts it.
The patch and its file are deleted.
U12. kube-prometheus-stack: kyvernoPolicyExceptions.enabled: false required on Kyverno-less clusters
The GS kube-prometheus-stack wrapper (22.0.0) renders a kyverno.io/v2alpha1 PolicyException for node-exporter's host access whenever node-exporter is
enabled. The template's API-version selection has a final fallback that
carries no .Capabilities.APIVersions.Has guard, so on a cluster without
Kyverno (this lab) the object renders anyway and the install fails on the
missing CRD. The lab needs node-exporter (node CPU/memory is half the point
of platform.observability), so the exceptions are switched off wholesale in
kube-prometheus-stack-values.yaml.tmpl — correct here regardless (no
Kyverno to except from), but the toggle exists only because of the missing
guard.
Unblocks: giantswarm/kube-prometheus-stack-app — guard the PolicyException
fallback with a Capabilities check like the sibling branches; then the value
can be dropped (it would render nothing here either way).
Since 2026-09-03 the bundled mcp-kubernetes, the managers (model-manager,
agent-manager, vm-manager, cluster-manager) and the lab's mcp-prometheus validate
the user's forwarded Dex id_token themselves (mcp-oauth resource servers against
global.identity), so each of them does OIDC discovery and JWKS fetches against
the issuer URL — https://localhost:<dexPort>/dex, the one URL the browser, the
apiserver and every pod must share (H-issuer, above). muster and Backstage reach
it through hostNetwork (U1); these cannot: all listen on :8080 and would collide
on the single kind node. Fix: a dex-localhost sidecar (socat, from its gsoci mirror) that
listens on the pod's own loopback : (IPv6 wildcard, dual-stack; the
IPv4 wildcard in a pod without an IPv6 stack, on a host that boots with
ipv6.disable=1, where the IPv6 socket cannot be created) and
forwards to the Dex ClusterIP Service, so localhost resolves inside the pod
exactly as on the host; Dex's certificate carries localhost, TLS verification
against the lab CA holds. Since 2026-09-08 a Kustomize strategic-merge patch in
components.<server>.postRenderers (postrenderers.go; the post-renderer
binary before), and the same patch on the lab's own mcp-prometheus HelmRelease
(mcp-prometheus.yaml.tmpl). Since 2026-09-16 the targets are a rule over the
component charts' renders, not a list: every Deployment whose containers are
told the lab Dex's localhost address (--dex-issuer-url=, DEX_ISSUER_URL) gets
the sidecar (dexLocalhostTargets), so a component an overlay turns on
(cluster-manager) or one the chart turns on by default is covered without a
lab release — the list had left both crash-looping on the unreachable issuer
(agentlab#58). model-manager, on by default in the chart since agent-platform
4.24.0 with no backend, never gets that far in a lab without one: the roster
states components.model-manager.enabled from platform.modelManager, so
the toggle off is the component off. agentlab platform-test
reads the same rule off the live Deployments and asserts the sidecar, a clean
rollout with no restart and the MCPServer Connected. Since 2026-09-17 the rule
keys on the names the servers dial the issuer by (--dex-issuer-url,
DEX_ISSUER_URL, OIDC_ISSUER_URL), not on any value carrying the address:
a resource server that pins the lab Dex as the authorization server it names
in its RFC 9728 metadata (OAUTH_AUTHORIZATION_SERVER) never dials it and had
been flagged by platform-test as a server missing its sidecar (agentlab#210).
The annotation agentlab.giantswarm.io/dex-localhost: "true"|"false" on the
Deployment or its pod template overrides the rule either way, and both
commands name the key each target was selected by (docs/platform.md).
The sidecar's securityContext meets PodSecurity restricted
(allowPrivilegeEscalation: false, capabilities.drop: [ALL],
seccompProfile: RuntimeDefault), so a rollout of a server prints no
PodSecurity warning. Lab-only by construction: real installations have a routable issuer.
platform.modelManager.backends lists every model server on the lab host
(agentlab configure fills it from what answers: an Ollama, a Lemonade
Server), but model-manager 0.16.0 fronts ONE backend per instance
(backend: ollama|kserve|lemonade). Timo's direction (agentlab#60,
2026-09-04) is one multi-backend model-manager, not one release per backend
(the draft that did that, #61, is closed). Until it lands, the lab installs
model-manager with the first backend and, for every further one, renders its
downloaded tool-calling models as lab-labeled ModelConfigs in exactly the
shape model-manager writes for that backend (lemonade-<model>: OpenAI on
<endpoint>/api/v1, placeholder key; ollama-<model>: the native keyless
provider), refreshed and pruned on every agentlab platform run and proven
by the agent turn models-test ends with. Nothing manages those models
(pull/delete stay CLI-on-host) and the portal's Models pages show the first
backend only.
Fix: model-manager 0.17.0 fronts several backends in one process
(--backends, GET /api/v1/backends, backend on every object and request,
one ModelConfig per (backend, model) carrying the
model-manager.giantswarm.io/backend label), the fleet's connectivity chart
3.7.0 and the umbrella (v0.35.15+) take model-manager.backends. The lab now
hands the whole list to model-manager.backends (a single entry renders the
backend: form), preflights every server, and hostmodels.go keeps only the
inventories (discovery summary, the delete check of models-test); the static
wiring, its pruning and its agent-turn proof are gone. models-test --backend lemonade proves the second backend through model-manager instead. The
agentlab.yaml shape did not change.
The umbrella's Backstage app-config renders app.baseUrl, backend.baseUrl
and backend.cors.origin as https://<hostname> — no port, because
components.backstage.hostname is also the HTTPRoute hostname and cannot
carry one. With platform.gatewayPort != 443 Backstage's OAuth redirect_uri
is port-free while the Dex client (dex.yaml.tmpl) registers the ported one,
and every login fails with "Unregistered redirect_uri". The lab's app-config
overlay (backstage-catalog.yaml.tmpl) restates the three URLs from
.BackstageBaseURL; at 443 the values are identical. In-cluster, the lab's
edge Service (gateway-nodeport.yaml.tmpl) serves that port as well, so the
ported URLs resolve from pods too (agentlab#67). Lab-only by construction: a
real installation runs its edge on 443, so the umbrella has no reason to
carry a ported public URL; the overlay is the lab's permanent answer, not an
interim.
A multi-image docker save -o <tar> a b c under Podman's docker-compatible
CLI writes ONE image carrying every name as a tag unless
--multi-image-archive is passed, so a batched archive lands one image under
all the tags (the Flux controllers crashlooped running flux-cli). Under podman
(runtime.go) the lab therefore saves and imports one image at a time
(kindLoadImages); under docker the batch is saved per platform (U21). The
save is the lab's own docker save now (the import is the embedded kind's
nodeutils.LoadImageArchive), so the lab could pass podman's
--multi-image-archive itself and batch there too — not done for lack of a
podman host to verify the flag through the compatible CLI on.
Rootless Podman publishes ports from the invoking user's network namespace,
where the kernel refuses everything below
net.ipv4.ip_unprivileged_port_start (1024). A bind probe alone reads a free
443 as usable, because the lab's own process cannot bind it either way, so
configure asks the engine instead (MinPublishablePort, runtime.go) and
ChooseFreePorts moves the edge to 8443. Resolves for a user who runs Podman
as root or lowers the sysctl; nothing to fix in the lab.
U18. oauth-fixture.yaml.tmpl: the sign-in fixture pins Dex as its authorization server — BLOCKED UPSTREAM
The fixture points muster at its own protected /mcp, whose RFC 9728 metadata
names muster's own OAuth 2.1 server. That server identifies muster-as-client by
Client ID Metadata Document, and mcp-oauth's SSRF guards refuse the metadata
URL — muster.127.0.0.1.nip.io resolves to the edge's cluster IP in-cluster
(the CoreDNS rewrite) and to loopback everywhere else (client_id metadata URL resolves to private/internal IP address … (SSRF protection)); dynamic
registration is refused the same way for the proxy callback's redirect URI
(hostname resolves to private IP address (DNS rebinding protection)). muster
exposes none of mcp-oauth's knobs for a lab (AllowPrivateIPClientMetadata,
DisableDNSValidation, AllowPrivateIPRedirectURIs), so no sign-in could
complete against it and the fixture was a challenge generator only. Fix in
the lab: the CR pins the lab Dex through spec.auth.authorizationServer
(Dex's authorizationEndpoint/tokenEndpoint, clientCredentialsSecretRef →
the platform client's id/secret in the Secret lab-oauth-fixture-client,
scopes — a pinned server gets no default scope and Dex refuses a request
without openid — and as issuer muster's own public URL, so the grant's
token-store key stays apart from muster's own login issuer (Dex) and a
core_auth_logout clears it; a pin naming Dex's URL as the issuer works since
muster 5.12.1 (muster#1174, muster#1175: the grant key follows the pin and a
changed pin takes effect without a restart) but leaves the grant behind on
logout, which would make the next run's sign-in proof reconnect without a
challenge), and dex.yaml.tmpl lists
muster's proxy callback on that client. Dex matches redirect URIs exactly and
the token it issues carries the platform client's audience, which the endpoint
trusts, so agentlab toolsets-test completes the sign-in headlessly. Unblocks
when muster exposes allowPrivateIPClientMetadata for its OAuth server — then
the pin can go and the challenge chain becomes proxy start → muster
/oauth/authorize → Dex again.
U19. components.agent-platform-connectivity.postRenderers: the kagent controller metrics Service selects the wrong instance label — FIXED upstream
The connectivity chart (up to 3.20.1) rendered a Service for the kagent
controller's metrics port and a ServiceMonitor on it, selecting the controller
pods with app.kubernetes.io/instance: <its own release name>. Under the
standalone umbrella every subchart shared one release name, so that matched;
under the meta chart kagent is its own release named kagent, the Service
selected no pod, and the lab Prometheus scraped nothing of kagent
(platform-test "Prometheus scrapes the platform itself" caught it on the
first meta-chart run). The lab carried a Kustomize strategic-merge patch on
the Service's selector in components.agent-platform-connectivity.postRenderers.
Fixed: agent-platform 3.20.2
(giantswarm/agent-platform#305,
PR #308) — the Service selects kagent's own release name. The patch is
deleted; the lab renders no postRenderers for the connectivity component,
and the lab's Prometheus scrapes the kagent controller through the chart's
own Service.
The kagent chart renders its workloads into kagent.namespaceOverride
(kagent), but its HelmRelease targets the release namespace like every
component, so helm-controller's createNamespace never creates kagent; the
one chart object that did — the connectivity chart's Namespace — sits in a
release that dependsOn kagent. A first install on a fresh cluster failed
every kagent attempt with namespaces "kagent" not found until the retries
were exhausted, and everything behind kagent waited (seen on the first
meta-chart run: 6 attempts, then Stalled). The lab created the namespace in
platformUp when agents are on, the way management clusters do in their
bases, and the connectivity release adopted it.
Fixed: agent-platform 3.20.2
(giantswarm/agent-platform#306,
PR #308) — with the bundled engine and kagent on, a pre-install,pre-upgrade
hook Job (<release>-kagent-namespace, weight -8) creates the namespace when
it is missing (and waits out a Terminating one left by a previous
uninstall) ahead of the kagent HelmRelease; the connectivity release adopts
it on install and deletes it with the release. ensureNamespace for kagent
is deleted; a fresh agentlab up gets the namespace from the hook.
U21. preload.go: the docker side-load is docker save --platform + kind's archive import — BLOCKED UPSTREAM
kind's load docker-image runs a plain docker save and pipes the archive
into the node's ctr images import --all-platforms. Under Docker's containerd
image store — the default on Docker Desktop and on new Docker 29 installs —
that archive carries a pulled image's whole multi-platform index while only
the host platform's blobs were ever pulled, and the import fails on the first
missing digest: failed to load image: command "docker exec ... ctr --namespace=k8s.io images import --all-platforms ..." failed with error: exit status 1 / content digest sha256:...: not found, once per side-load lane,
every image degrading to the in-node pull the preload exists to avoid
(kubernetes-sigs/kind#3795,
open since 2024-11; the maintainers point consumers to docker save --platform | kind load image-archive and will not lock the platform inside kind load).
The lab does exactly that, with kind embedded (kind.go): the load path of
kind's load docker-image command is not used at all — dockerLoadImages
asks docker image inspect which platform the host holds each ref in, runs
one docker save --platform <p> per platform and streams its stdout into
the node's ctr images import — the command of nodeutils.LoadImageArchive,
the library call behind kind load image-archive, minus its --digests
(loadNodeArchive): that flag names every imported manifest a second time
import-<date>@sha256:…, which the CRI lists as docker.io/library/import-…,
a name containerd does not have, and a container resolved to it fails with
"failed to check if this is a checkpoint image". Streamed, not staged: kind load docker-image writes the
archive to the temp dir first, and the archive of everything a lab ran is
tens of GiB — on /tmp, a tmpfs on many Linux hosts, that is RAM for the
length of the import, and for good when the boot is interrupted (a 20 GiB
leftover was found filling a host's swap). Needs Docker 28 (API 1.48) for
docker save --platform; an older client/daemon gets one plain save of the
batch, which is right under the classic graph driver. Podman keeps its
one-save-per-image load (U16).
Unblocks when kind's load docker-image logic (the re-tag of an image ID the
node already has, the per-image save) survives the containerd image store and
is worth reusing over the plain archive import.
U22. substratepools.go: Substrate's CA/JWT pool bootstrap is a Go port of kubectl-ate — FIXED upstream
The substrate chart (0.0.26) mounts four pool Secrets, a trust-anchor Secret
and an authentication ConfigMap it does not render, and upstream's install was
helm install → kubectl-ate admin make-ca-pool / make-jwt-pool plus a
shell step → a second helm upgrade --wait. While no chart shipped Substrate
the lab installed it itself ahead of the platform, with substratepools.go
copying the generate + serialise subset of substrate's localca and
localjwtauthority packages and substrate.go creating every bootstrap
object before one waited install (the POC channel's shape).
Fixed upstream (agent-platform 4.0.x): the meta chart ships Agent
Substrate as the substrate-crds and substrate components, and the
connectivity release's pre-install,pre-upgrade hook Job
<release>-substrate-bootstrap mints the same key material (openssl in an
init container, the pools' wire format, a pool that exists is never touched)
and publishes the trust anchors — the lab's port was that Job's prior art.
The lab installs nothing of Substrate anymore: substrate.go keeps the one
check the chart cannot make early — the apiserver serves
certificates.k8s.io/v1beta1, else agentlab down && agentlab up — and the
platform.substrate knob, the Go port, its tests and the Substrate values
template are gone (agentlab#138). The chart's substrate component adopts a
lab's earlier substrate/substrate-crds releases in ate-system by name
(storageNamespace = targetNamespace), so a lab that ran the POC channel
upgrades in place.
LM Studio's own API (/api/v1, 0.4.0+) serves the library, the download that
backs a pull, and load/unload — but no delete. Removing a model is lms rm
on the host, a CLI no pod can run, so the lmstudio backend of model-manager
reports delete: false and the platform answers 501 unsupported.
Consequence in the lab: agentlab models-test --backend lmstudio is the
one proof run that leaves something behind — the model it pulls stays
downloaded, and the run's last line says so with the lms rm command. Rather
than skip the step, the run asserts the refusal (a stronger check: the
platform must refuse rather than pretend, nothing may be removed by a refused
delete, and the ModelConfig must still come off through POST /models/unwire), and it cross-checks the advertised delete capability
against what the server really offers in both directions. Unblocked by an
LM Studio release that exposes over its API what lms rm does; the lab side
is then one deleteOverREST: true in internal/lab/backends.go plus the
driver's capability flag.
Every other host model server answers a version or health document, so the
discovery could ask "HTTP 200 with a version field?". LM Studio has no
version, health or system-info endpoint anywhere, and it answers HTTP 200
with an {"error": …} body for every path outside its own /api/v1 —
Ollama's /api/version, /api/tags and /api/show among them (verified
against 0.4.20; paths under /api/v1 do 404). A probe that trusted the
status code would therefore find an "Ollama" on every LM Studio port.
Fix: detection is a per-backend fingerprint over the response body
(backendProbe in internal/lab/backends.go) — LM Studio is recognised by
an /api/v1/models document carrying a models array whose entries are
keyed by key, which also distinguishes it from Lemonade serving the very
same path with a data envelope. The discovery prints api v1 where the
others print a version. Not a workaround to remove: the body is the only
evidence such a server offers. The unit tests carry the 200-on-unknown-path
behaviour in the fake, so a future "simplification" back to a status check
fails them.
U25. helm.go: a schema refusal is classified by re-validating, not by Helm's error type — BLOCKED UPSTREAM
A render the values fail is the one render failure that predicts the install
(schemaRejection; fluxreleases.go refuses the install on it), so the lab
must tell it from every other render error. Helm 4.2 does type the verdict —
ValidateAgainstSingleSchema returns JSONSchemaValidationError for a
validation verdict and a plain error for a schema that does not unmarshal or
compile — but ValidateAgainstSchema walks the dependency tree and flattens
every chart's verdict into one errors.New(sb.String())
(pkg/chart/common/util/jsonschema.go), and ToRenderValuesWithSchemaValidation
wraps that with %w: the type never reaches Install.RunWithContext's
caller, so errors.As cannot find it. schemaRejectionOf therefore re-runs
the same coalesce (CoalesceValues) and asks each chart's schema down the
tree itself (ValidateAgainstSingleSchema), on the error path only and only
for a render error carrying Helm's schema headline (values don't meet the specifications of the schema(s)) — a second coalesce and schema compile per
refused render, and a second fetch of a remote $ref where a schema has one.
Unblocks when ValidateAgainstSchema keeps the verdicts' type on the way out
(an errors.Join of the per-chart JSONSchemaValidationErrors, or a typed
aggregate); the re-validation then becomes one errors.As on the render
error.
U26. postRenderers patch: the klaus-gateway component's Slack Web API is the proof's fake — ACCEPTED
klaus-gateway is Slack-only since 2.0 (giantswarm/klaus-gateway#319) and no
workspace answers the lab, so the component's Slack adapter has nothing to
talk to, and its chart has no value for the Web API base the gateway reads
(KLAUS_GATEWAY_SLACK_API_BASE, --slack-api-base). Fix: a
strategic-merge patch in components.klaus-gateway.postRenderers
(slackAPIPatch, postrenderers.go) sets it to the selector-less Service
agent-platform/agentlab-slack-api, which klaus-gateway-test points at its
fake Slack Web API while it runs, and removes afterwards. The fake then runs
as a container on the kind network (the proof's own static binary,
agentlab slack-fake, in the lab's probe image) rather than on this host:
pods reach a container on the kind network directly, while a service on the
host is behind the host's firewall, which a default-deny one closes to
them. Outside a proof nothing answers there and nothing calls it: the
Events API adapter calls the Web API only for an event, and the proof is
the only sender. klaus-gateway-test asserts the patch on the live
Deployment and fetches the fake through the Service from a probe pod first.
Lab-only by construction.
U27. hostkernel.go: the node's kubelet and systemd-sysctl would rewrite the host's kernel settings — BLOCKED UPSTREAM
The kind node is a privileged container, and under a rootful engine its root
is the host's root, so a kernel-global sysctl written in the node lands on
the host (#309). At every node start (up and every host
boot) kubelet's setupKernelTunables sets vm.overcommit_memory=1,
vm.panic_on_oom=0, kernel.panic=10, kernel.panic_on_oops=1 and the
root key quotas, so a kernel oops reboots the workstation. The node's
systemd-sysctl applies the node image's Debian /usr/lib/sysctl.d, where
kernel.core_pattern=core switches off the host's systemd-coredump. kubelet
has no mode that leaves the host alone: protectKernelDefaults refuses to
start on any host whose values differ. Fix: the kind config mounts a
kubelet drop-in whose ExecStartPre bind-mounts a shadow file carrying
kubelet's value over each of its six /proc/sys keys inside the node, so
kubelet reads what it wants and writes nothing. It also mounts an empty
directory over /usr/lib/sysctl.d, which leaves systemd-sysctl only kind's
own /etc/sysctl.d (per-network-namespace net.* keys). Pods mount their
own /proc and see the host's values. up compares the host's values
before and after it creates the node and warns with the restore command.
On an existing node created without the mounts it says to recreate the node.
Unblocks when kind stops applying the image's sysctl.d
(kubernetes-sigs/kind, as with its systemd-binfmt mask) and kubelet gains a
mode that neither writes nor refuses. A rootless engine needs none of it, since
the kernel refuses kernel-global writes from a user namespace.
U28. The BYO issuer: an /etc/hosts of its own for the apiserver, a CoreDNS rewrite for the pods — ACCEPTED
With an externally provisioned pair (platform.tls) the issuer is
https://dex.<domain>:<dexPort>/dex (#20), and the name
must reach Dex from the apiserver, which reads the issuer at boot and
resolves through the node's resolver, and from every pod. kind exposes no
--add-host for its node, and the kubelet copies the node's /etc/hosts
into a hostNetwork pod at its start, so an entry written into the node after
kind create misses the apiserver. Fix: certs/apiserver-hosts
(dex.<domain> on loopback, where the Dex NodePort answers) mounted over the
apiserver's /etc/hosts through the ClusterConfiguration's extraVolumes
— a container that mounts /etc/hosts gets no kubelet-made one — and, for
the pods, a CoreDNS rewrite of dex.<domain> to the dex-issuer ClusterIP
Service on the issuer's port, ahead of the wildcard to the edge. The pair
covers no in-cluster name, so the agentgateway JWKS sources dial the issuer's
host too, and the dex-localhost sidecar (U13) is not patched. Inherent to
one issuer URL on a single-node kind cluster; nothing upstream to wait for.
go install github.com/giantswarm/agentlab@latest refuses a module whose
go.mod carries a replace or exclude directive, and the weekly nancy
remediation moved two modules nobody compiles in with exactly that
(#388; the class #36 fixed before): certificate-transparency-go
v1.3.3, the newest release, requires prometheus/prometheus v0.51.0 and
etcd/v3 v3.6.8, which nancy flags from the module graph. A plain require
of the fixed versions does not stick — neither module provides a package the
build imports, so go mod tidy prunes it again. Fix: the dependency is
bumped to the master commit that drops both (the Trillian v1.8.0 bump,
2026-09-21), the one upstream version without them, so the graph is clean
with no .nancy-ignore; gomod_test.go fails on any replace/exclude in
go.mod, so the next remediation of that shape goes red instead of merging.
Unblocks when certificate-transparency-go releases after v1.3.3: a rule in
renovate-custom.json5 proposes that tag as the bump PR, which removes this
entry.
Agent Substrate registers the CSI driver that mounts a workspace's volume
into an actor through its CSIDriverConfig (the controller endpoint, the node socket
override, the mTLS settings), which the agent-platform chart renders from
its workspaces.substrate.csiDriver values — agent-platform 4.127.0 and
newer. Releases before take no workspaces block, and 4.125.0 to
4.127.0-rc.2 take the block and render nothing from it, so taking the key is
not registering the driver. Fix: ensureWorkspacesCSIDriverConfig
applies the lab's own CSIDriverConfig nfs.csi.k8s.io (labelled
app.kubernetes.io/managed-by=agentlab) after the platform install whenever
the component renders the install applies carry none
(rendersWorkspacesCSIDriverConfig reads them, the render that failed
counts as none); a Helm-managed object still live is an earlier chart's
that helm-controller's upgrade removes, waited out before the lab applies
its own (one that outlasts the wait stays the chart's), and one applied by
hand becomes the lab's. Before the install of a chart
whose render carries it, the lab's object is removed
(removeLabWorkspacesCSIDriverConfig: Helm adopts no object another
manager created). The values template renders the workspaces: block for
every chart that takes it (chartCarriesWorkspaces). Unblocks when the
oldest chart the lab supports renders the CSIDriverConfig: the
lab-applied object and this entry go.
U31. workspacegrant.go: the lab signs the workspace grants and wires kagent's admission — BLOCKED UPSTREAM
A Session on a workspace volume is admitted by kagent's jwt Session
admission against the workspace-manager's key set, with a grant the manager
signs. The platform renders neither half: the manager's grant.signingKey
names a Secret nothing creates, so it serves no authorize_workspace and no
/.well-known/jwks.json, and kagent's controller.session.admission.jwt is
unset, so every Session that names a volume is refused. Fix: with the
workspaces and the agents on, the lab generates an EC P-256 key once
(certs/workspace-grant/grant.pem), places it as the Secret
agent-platform/workspace-manager-grant ahead of the install and names it in
workspace-manager.grant.signingKey. It points kagent's admission at the
manager's key set through its Service, with the manager's base URL as the
issuer and kagent as the audience, and adds the claude Harness the proof's
Session runs on (kagent.harnesses). workspaces-test signs its grants with
the same key in the manager's claims, so they verify through the key set the
manager publishes. Unblocks when the platform creates the signing key
(agent-platform#935)
and wires kagent's admission to it
(agent-platform#890).
Then the values template drops the lab's key and admission, and the proof's
grant comes from authorize_workspace once the manager signs the identity
kagent admits
(workspace-manager#36).
A claude Session works in its volume directory from kagent 1.9.0
(agent-platform#936).
On an older kagent line, the proof's Session on the volume reports its default
directory and fails.
U33. agent-platform-values.yaml.tmpl: the lab lists tenants-test's namespaces in Substrate's credential policy — BLOCKED UPSTREAM
With platform.tenants on, an agent in an organization namespace needs
Substrate's credential provider to read that namespace's Secrets (its model
key), and the chart's policy names the kagent namespace only. Fix: the
lab's values set substrate.credentialProvider.namespacePolicies to the
chart's two entries plus tenants-test's two namespaces
(tenantsTestNamespaces, tenantstest.go): each read by the atespace of its
own name, and by ate-golden. Other organization namespaces of the lab get
no credentials. Unblocks when the chart selects the organizations'
namespaces by label
(agent-platform#1041,
Substrate 1.12 with giantswarm/substrate#456): the chart then adds the
selector entries to the lab's list, tenants-test checks the provider's live
grants, and the list is redundant — the values template drops it.
The component dials muster at its public URL (obo.musterUrl): every Slack
Session starts through agent-manager's MCP tools over muster, the workspace
listing and the OBO discovery go there too. Inside a pod that URL is the edge,
whose certificate the lab CA signs, and the chart has a CA value only for a
grpcs:// A2A target, so the turn fails with x509: certificate signed by unknown authority. Fix: a strategic-merge patch in
components.klaus-gateway.postRenderers (labCAPatch, postrenderers.go)
mounts the lab's trust bundle, Secret agent-platform/dex-ca (the one muster,
the portal and workspace-manager trust), read-only at
/etc/klaus-gateway/lab-ca and names it as SSL_CERT_FILE, the system roots
Go reads; the image's own roots stay, read from its certificate directory.
klaus-gateway-test asserts the patch on the live Deployment and names the
muster URL and the CA source in the in-cluster turn's error. Lab-only by
construction: an installation's edge serves a publicly trusted certificate.
U35. tenantstest_slack.go: on macOS the Slack steps run the gateway's image on the kind network — ACCEPTED
tenants-test's Slack steps run klaus-gateway as klaus-gateway-test does,
the image on the host's network. Docker Desktop leaves host networking off by
default, so the Mac has no route to the container's loopback ports and the
readiness probe times out with an empty log; a darwin build of the gateway
cannot verify the lab CA either (SSL_CERT_FILE is not read on macOS, and the
system verifier calls the lab certificate not standards compliant). Fix:
on darwin, with the image, the container joins the kind network, its two
ports are published on 127.0.0.1, --add-host names the edge's and muster's
public hostnames as the node's address, the target and muster are the edge's
NodePort (config.GatewayNodePort) under those names (TLS verifies as on the
host), and the fake Slack Web API is host.docker.internal. Linux keeps the
host network. The bridged shape is tenants-test's only; klaus-gateway-test
is unchanged.
U36. tenantstest_slack.go: an allowed collaborator sends again after the replay note — BLOCKED UPSTREAM
After the thread's person allows a collaborator who may not use the thread's
agent, klaus-gateway 4.10.0-rc.1 replays the held message, the controller
refuses it, and the replay path answers with its generic "Your message could
not be picked up again. Send it again." (postReplayFailureNote knows only a
share refusal), not the note naming the agent's tenant. Fix: the proof
records the note and sends the collaborator's message again; that turn gets
the refusal, and the verdict says so. Unblocks when the gateway's replay note
uses its not-permitted note (giantswarm/klaus-gateway, postReplayFailureNote
in pkg/channels/slack/slack.go): the resend then goes.
U37. tenantstest_edge.go: the editor's UpdateAgent is tried again while it answers Internal — BLOCKED UPSTREAM
Right after the editor's CreateAgent, kagent 1.19.0's UpdateAgent reads
the Agent from the controller's cache, which has not seen the create's own
status write yet, and the API server's conflict comes back as Internal
(kubecrud.Service.Update maps every error but Invalid to Internal). Fix:
step E5 tries the update again while it answers Internal, for up to 30 s
(edgeUpdateConflictWait); any other answer, or Internal after that, fails
the step. Unblocks when kagent's Update maps a conflict to Aborted or
retries it on a fresh read (giantswarm/kagent-upstream,
go/core/internal/service/kubecrud): the retry then goes.
agentgateway binds its readiness (15021) and stats (15020) servers and its
listeners on the IPv6 wildcard unless IPv6 is switched off
(giantswarm/agentgateway-upstream v2.1.2, the 2.1.2 image's git_revision
39023841: crates/agentgateway/src/config.rs:52-68 — IPV6_ENABLED or
config.enableIpv6, default true, picks bind_wildcard;
config.rs:363-369 the readiness default; READINESS_ADDR/readinessAddr
would move only that one server). On a host that boots with
ipv6.disable=1 no pod has an IPv6 stack, so the edge's data plane (and
Substrate's atenet router and egress, agentgateway 2.2.1+) crash-loop on
readiness server starts — Address family not supported by protocol (os error 97). agentgateway does check for a disabled IPv6, but only lo's
/proc/sys/net/ipv6/conf/lo/disable_ipv6 (lib.rs:981-991), only for the
localhost addresses, and treats the file being absent — exactly the
ipv6.disable=1 case — as IPv6 on. Fix: patch 9 of postrenderers.go:
rawConfig.config.enableIpv6: false on every AgentgatewayParameters the
component charts render (the edge's, the models Gateway's) and
IPV6_ENABLED=false on every container that runs the data-plane image,
selected off the renders (agentgatewayIPv4Targets). On every host: the
kind cluster is IPv4 single-stack, so nothing reaches the data plane over
IPv6 anyway. Unblocks when agentgateway falls back to the IPv4 wildcard on a
host without IPv6 (an absent /proc/net/if_inet6, or EAFNOSUPPORT from
the IPv6 socket) — then the patch goes.
- No HSTS from the portal — Backstage's helmet sends
Strict-Transport-Security: max-age=15552000; includeSubDomainsand has no setting for it; a browser that trusted one lab CA then refuses the hostname for 180 days with no way past the warning once the CA changes (a recreated lab, another domain's re-mint). The lab's edge drops the header on the connectivity chart'sbackstageroute (postrenderers.go, patch 8);backstage-testasserts it is gone. A machine-local CA has nothing HSTS protects. - Checksum stamping via the
REPLACED_AT_APPLYplaceholder — the standard Helmchecksum/configpattern, done with sed because there is no templating engine here. Deduplicated into one script (H3), otherwise kept. - Password grant + static secrets in git (
kubernetes-lab-secret, bcrypt of "password", the Backstage session key) — the lab's identity is self-contained and throwaway by design; nothing here guards anything real. skipTLSVerify: truefor the Backstage→apiserver hop — the alternative is minting the kind CA into the Backstage trust store; zero value in a lab.test-backstage.shregex-parsingdecodeURIComponent('…')out of the login response — it drives a browser postMessage flow headlessly; there is no API that returns this payload cleanly. Inherent to the test's job.vendor/agent-platform-standalone/hack/curate.sh— upstream's file, vendored and gitignored; out of scope here.login-browser.pyfixed callback port 5555 — must be pre-registered in Dex'sredirectURIs; a random port would break the static client. By design.- The Cilium policy CRDs served without Cilium (
cilium.io/v2,internal/lab/templates/cilium-crds.yaml, applied byupnext to the Gateway API CRDs;providedapis.go) — every Giant Swarm cluster runs Cilium, so component charts renderCiliumNetworkPolicyobjects unconditionally (gpu-operator-app) and their HelmReleases fail in kind without the API. The alternative, a Cilium in kind, would replace kindnet for the sake of policies the lab never tests; the CRDs alone let the charts install and validate their policies against the real schema, and enforce nothing. Named as a lab-only difference in docs/platform.md. - The fleet's
flux-multi-tenancypolicy on the fleet's Kyverno charts reduced to the admission controller, the platform namespace exempt in code (internal/lab/admission.go,templates/flux-multi-tenancy.yamlbyte-identical to management-cluster-bases,templates/kyverno-values.yaml,templates/org-fixture.yaml) — the lab installs what security-bundle installs,kyverno-crds1.17.0 and the kyverno app 0.24.2 (the upstream chart 3.7.2 = v1.17.2 and policy-reporter) fromgsoci.azurecr.io/charts/giantswarm, with values that turn off what a one-node kind cluster has no API or use for (CiliumNetworkPolicy, VerticalPodAutoscalers, PodLogs, ServiceMonitor, HPA, the app's own policies and PolicyExceptions, every controller but admission). policy-reporter is a dependency without a condition, so its one Deployment runs too. The exemption ofagent-platformis the lab's one deviation from the policy text: on an installation the platform's child HelmReleases live in the exemptflux-giantswarm, in the lab shape in the platform namespace, and rewriting the fleet's YAML by hand would hide that the file is the fleet's — so the namespace is appended to every rule's exclude list at apply time and the embedded file stays refreshable from the source. Named in docs/platform.md "Lab-specific deviations". - The OAuth sign-in fixture aggregates muster itself (
lab-oauth-fixture→ muster's own protected/mcp,internal/lab/oauthfixture.go) — the one way to get a downstream that staysAuth Requiredbehind an authorization server that accepts muster's CIMD client id without adding a workload (Dex knows only static clients). Costs:Failedfor about a minute after a muster restart (self-dial before the listener is up;agentlab platformwaits it out), and a completed sign-in connects muster to itself. Both accepted; see docs/platform.md "Signing in to a downstream server". - A twice-replaced, once-trusted CA lingers only until the next trust op —
a
platform.domainchange stashes the outgoing CA undercerts/replaced/, and bothagentlab trustanduntrustsweep every stashed CA out of the trust stores before doing their own work. The stash (not the store) is the source of truth for what to remove, because the stores index roots by name+serial and the serial dies with the overwrittenca.crtotherwise.
With every fix above in place, one full cycle on a cold cluster:
| Step | Result |
|---|---|
make up (fresh kind cluster, no apiserver bounce) |
issuer up, apiserver accepts Dex tokens |
make test |
10/10 RBAC assertions pass for admin/dev/viewer |
make platform |
deps rebuilt via digest gate, MCPServer Connected, muster live on :8090 |
make platform-test |
Dex → muster → mcp-kubernetes → apiserver chain passes |
make backstage |
image loaded once (exact-tag check), pod up |
make backstage-test |
all three users sign in, reach muster, see workflows/tools |
make backstage re-run |
no image reload, same pod, same revision (checksum no-op) |
make reload |
no-op apply, Dex stays at revision 1 |
make down |
cluster deleted, no leftovers |
The same cycle through agentlab, on a cold cluster, defaults from
agentlab configure --defaults:
| Step | Result |
|---|---|
agentlab configure --defaults / --platform --backstage |
agentlab.yaml written, bcrypt hashes cached |
agentlab up (fresh kind cluster) |
issuer up, apiserver accepts Dex tokens |
agentlab test |
10/10 RBAC assertions pass for admin/dev/viewer |
agentlab login dev@lab.local |
kubeconfig.oidc works, kubectl auth whoami = oidc:dev@lab.local |
agentlab platform |
deps built via digest gate, MCPServer Connected, muster live on :8090 |
agentlab platform-test |
Dex → muster → mcp-kubernetes → apiserver chain passes |
agentlab backstage |
image loaded once (exact-tag check), pod up |
agentlab backstage-test |
all three users sign in, reach muster, see workflows + 29 core tools |
agentlab backstage re-run |
no image reload, same pod (checksum no-op) |
agentlab reload (unchanged config) |
no-op apply, single ReplicaSet |
agentlab up re-run (components enabled) |
idempotent: cluster reused, secrets kept, post-render patches survive the helm upgrade |
edit a user in agentlab.yaml + agentlab reload |
pod rolls (checksum), immediate login as the new user succeeds (H11) |
custom config (agentlab2, Dex :31000, run from an empty dir) |
second cluster up alongside the first, 10/10 RBAC, clean down |
agentlab down |
cluster deleted, no leftovers |
Form coverage: go test ./... drives the real huh form with scripted
keystrokes (accept defaults, edit fields, toggle components) and unit-tests
the post-renderer against a synthetic Helm release.
On an SELinux-enforcing host (Fedora) under a rootful engine (Podman through
sudo), a container's process may not read or execute a bind-mounted file
labelled user_home_t, and every lab container bind-mounts one: the fakes
(the skill host, the fake Slack Web API, the mockllm) run agentlab's own binary
from the user's home, beekeeper and klaus-gateway mount a run directory and
the lab CA. The container exits at once and the step that started it stops with no
log. Fix: dockerRun passes --security-opt label=disable to every
docker run. Relabelling (:z) would change the user's own files and is
undone by the next self-update; the lab's containers are throwaway
processes of the user's own binary and images, so the separation guards
nothing the user does not already own.