Skip to content

feat(harness): caller routes on the Claude harness forwarder, git through them - #126

Draft
QuentinBisson wants to merge 154 commits into
giantswarmfrom
feat/claude-harness-git-leg
Draft

QuentinBisson wants to merge 154 commits into
giantswarmfrom
feat/claude-harness-git-leg

Conversation

@QuentinBisson

@QuentinBisson QuentinBisson commented Sep 28, 2026 •

Copy link
Copy Markdown

Problem

A coding agent must git clone and fetch as the person who sent the turn, with no GitHub token in the sandbox (bumblebee-plans#60, D9 phase 2). The #103 forwarder carries the caller's bearer to MCP servers only; git has no path that acts as the caller.

Change

KAGENT_CLAUDE_CALLER_ROUTES (requires KAGENT_PROPAGATE_TOKEN=true) maps a host to an upstream that acts on the caller's credential for that host; in the platform, an agentgateway route trades the bearer at muster's broker for the person's token and refuses push on day one (giantswarm/agent-platform#754). The forwarder serves each host on loopback under /route/<host>/ only while a turn runs (401 otherwise) and ends a turn's requests in flight when the turn ends, refuses dot segments and encoded slashes before matching, streams bodies without the MCP limit, and git reaches it through GIT_CONFIG_* insteadOf scoped to the routed hosts. Proven on agentlab 2026-09-28 (bumblebee-plans coding-agents/poc/RESULTS.md).

Towards giantswarm/giantswarm#38056.

@QuentinBisson
QuentinBisson force-pushed the feat/claude-harness-caller-token branch from 11cfcd7 to 8473cf3 Compare September 29, 2026 19:43
@QuentinBisson
QuentinBisson force-pushed the feat/claude-harness-git-leg branch from 8b5adef to eab3e8b Compare September 29, 2026 19:45
@QuentinBisson
QuentinBisson force-pushed the feat/claude-harness-git-leg branch from eab3e8b to bfab23a Compare October 1, 2026 12:20
@QuentinBisson
QuentinBisson force-pushed the feat/claude-harness-caller-token branch from 0db83ec to c1d074b Compare October 1, 2026 12:36
Base automatically changed from feat/claude-harness-caller-token to giantswarm October 1, 2026 13:11
@QuentinBisson
QuentinBisson force-pushed the feat/claude-harness-git-leg branch 3 times, most recently from 3829cde to f1e4be1 Compare October 1, 2026 15:57
@QuentinBisson
QuentinBisson marked this pull request as ready for review October 1, 2026 16:04
@QuentinBisson
QuentinBisson requested a review from a team as a code owner October 1, 2026 16:04
@QuentinBisson

Copy link
Copy Markdown
Author

Superseded by #147: the egress gateway sets the caller's token through a kagent credential provider, so no loopback route, env var or agentgateway route is needed.

@QuentinBisson QuentinBisson reopened this Oct 2, 2026
@QuentinBisson
QuentinBisson marked this pull request as draft October 2, 2026 10:27
@QuentinBisson
QuentinBisson force-pushed the feat/claude-harness-git-leg branch from f1e4be1 to 40f736c Compare October 2, 2026 10:27
@QuentinBisson

Copy link
Copy Markdown
Author

Parked: git as the person is postponed. Upstream Substrate is redesigning this area (actor JWT injection merged in agent-substrate/substrate#1960, token exchange of the actor JWT in agent-substrate/substrate#1661, the actor JWT contract in agent-substrate/substrate#1756), and this fork line has to re-pin past agent-substrate/substrate#1809 before it can follow. This route design is not the path for the pilot. Kept as a draft so it can resume in place; tracking in giantswarm/giantswarm#38064.

@QuentinBisson
QuentinBisson force-pushed the feat/claude-harness-git-leg branch from 40f736c to b55bc99 Compare October 2, 2026 15:14
teemow and others added 10 commits October 5, 2026 14:21
…n label

helm-controller installs OCI charts under <version>+<digest>; '+' is not a
valid label value character, so every object of the chart failed to apply
under Flux. Sanitised the way helm.sh/chart already is.
…oc branch

The fork's tag.yaml publishes the dev builds of the agent-platform's kagent
component: GitHub-hosted runners (the org has no self-hosted runners), the
registries derived from the repository owner (ghcr.io/<owner>/kagent/<image>,
oci://ghcr.io/<owner>/kagent/helm/<chart>), no PyPI/GitHub-release jobs, the
image matrix trimmed to what the chart references plus the Claude harness
(multi-arch kept), and a push to poc/agent-platform that derives the version
as <base>-dev.<branch>.<committer date>.<time>.h<sha7> — the schema
giantswarm's gitsemver emits, so Flux selects the channel with a semverFilter
on the branch part. workflow_dispatch keeps its optional version input.

The chart's defaults are stamped with the published version and image
repositories before packaging: the tag must be explicit because the chart's
fallback, .Chart.Version, carries `+<digest>` under helm-controller.

The Makefile's -X targets followed $(DOCKER_REPO); with an owner-relative
repository they missed the module, leaving the binaries without version
information. They now follow the Go module path from go.mod.
…failure

apk.cgr.dev intermittently answers some of the parallel anonymous package
fetches from hosted runners with HTTP 403, which apk reports as
"Permission denied" for a random package and which fails the wolfi-based ui
image while the alpine-based images pass. The builder keeps the finished
stages, so the step now retries the build up to three times, repeating only
the failed step; and one image's failure no longer cancels the other three.
The controller and Go ADK images are alpine:3.22 plus `apk add`; the base
layer's libcrypto3/libssl3 3.5.7-r0 carry CVE-2026-14456 (HIGH, fixed in
3.5.8-r0). `apk upgrade --no-cache` before the add brings the runtime layer to
the current Alpine packages, as kagent-dev#2756 did for the harness images.
…w, keep a ledger

The consumed branch was built and published without a test or a scan:
upstream's ci.yaml and image-scan.yaml trigger on main and release/** only,
and their jobs ask for Blacksmith runners the org does not have. Both now run
on every push to poc/agent-platform and sync/**, on every pull request to
poc/agent-platform and (the scan) weekly, on GitHub-hosted runners with a
docker/setup-buildx-action builder, with the image matrices trimmed to what
the fork publishes (controller, ui, golang-adk, claude-harness — the Python
ADK, the Codex harness and the CLI image are not offered on the platform), a
govulncheck job over the Go module graph, no paths-ignore (a docs-only pull
request must still report the required checks) and one aggregate status per
workflow — ci-ok, scan-ok — for the ruleset to require.

sync-upstream.yaml re-pins the line weekly and on demand: fast-forward the
mirror main to upstream main with the workflow token (no workflow reacts, so
upstream's CI never runs on the mirror), rebase the carried commits onto the
upstream head (scripts/fork/repin.sh, which also names commits upstream merged
in the meantime and drops them), push the candidate with the bot token so the
checks run, wait for ci-ok and scan-ok, then force-push the consumed branch
with --force-with-lease so tag.yaml publishes the dev build. A conflicting
carried commit or a red check stops the run and opens a pull request against
the consumed branch that names the cause; nothing else is pushed.

tag.yaml gains a record job: every published build's version, commit, upstream
pin and image and chart digests go into the run summary and into builds.md on
the ledger branch (scripts/fork/ledger-append.sh; re-pins get the same on
re-pins.md). FORK.md is the human half of that ledger — the pin, every carried
commit with its purpose and upstream state, what is published and how, the
automated and the manual re-pin, protection, the contribution process — and
the README opens with it.
The first sync run was refused at the mirror: "refusing to allow a GitHub App
to create or update workflow `.github/workflows/ci.yaml` without `workflows`
permission" — the workflow's own token may not update a workflow file, not
even in a mirrored upstream commit, and no `permissions:` grant changes that.
The mirror is therefore pushed with the heraldbot App's token like the
candidate and the consumed branch. An App push starts upstream's workflow files
at the mirrored refs (runners the fork does not have, registries it must not
touch), so mirror.sh records what it pushed and mirror-quiet.sh cancels those
runs right after; the ledger stays on GITHUB_TOKEN.
…arent

The second sync run reached its "checks red" step and failed there:
`pull request create failed: GraphQL: Resource not accessible by
integration (createPullRequest)`. In a checkout of a fork, gh resolves
the base repository to the parent (kagent-dev/kagent) unless --repo is
given, and the App is not installed there. Both `gh pr create` calls now
name this repository.

git-push-as.sh emits its `::add-mask::` line only under GitHub Actions:
run by hand (the manual re-pin) it printed the encoded credential to
the terminal instead of masking it.
…warm line

SUBSTRATE_REPO defaults to oci://ghcr.io/giantswarm/substrate/helm and
SUBSTRATE_VERSION to 0.0.27-dev.giantswarm.2026-09-10.19-33-37.h734ec53: the substrate and substrate-crds subcharts of
kagent and kagent-crds are the fork giantswarm/substrate's builds (upstream
v0.0.26 — the version go/go.mod pins — plus cherry-picked fixes). The Go
module pin stays upstream's release, so the version is set here rather than
derived from go.mod; both move together at a re-pin.

Signed-off-by: Timo Derstappen <timo@giantswarm.io>
The Makefile's SUBSTRATE_VERSION / SUBSTRATE_REPO are the one pin: a
substrate-pin target prints them, the e2e job reads them into its
environment and installs the line's substrate-crds and substrate charts and
its ateom-gvisor worker image (the fork's tags carry no v). kubectl-ate for the
CA/JWT bootstrap stays upstream's v0.0.26 binary — the line publishes no CLI.
Before, the job exported SUBSTRATE_VERSION=0.0.26, which also drove make
helm-version and made the fork registry look for a chart it never published.

Signed-off-by: Timo Derstappen <teemow@gmail.com>
The consumed branch was renamed from poc/agent-platform to giantswarm (the GitHub rename retargets the default branch, the ~DEFAULT_BRANCH ruleset and open pull requests); every reference follows: the push and pull-request triggers of ci.yaml, image-scan.yaml and tag.yaml, CONSUMED in sync-upstream.yaml and scripts/fork/repin.sh, FORK.md and the README. The dev version's branch part becomes 'giantswarm' — 0.11.0-dev.giantswarm.<date>.<time>.h<sha7> — and the consumers' semverFilter moves with it (.*-dev\.giantswarm\..*).
teemow and others added 20 commits October 8, 2026 09:00
Signed-off-by: Timo Derstappen <teemow@gmail.com>
TestServerTelemetryShape read the span exporter and the metric reader
right after GetVersion and Check returned. The client has its response
before the server is done with the RPC: otelgrpc ends the server span
and records rpc.server.call.duration on the stats handler's End event,
after the response is written, so either could still be missing and the
test failed with "rpc.server.call.duration was not recorded".

Wait, bounded, for the GetVersion server span and the recorded metric
before asserting; the assertions are unchanged.

Signed-off-by: Timo Derstappen <teemow@gmail.com>
Signed-off-by: Timo Derstappen <teemow@gmail.com>
Every Pause, Suspend and fencing Resume a quiescence holder sends carries
a fencing token. A Substrate whose Actor API predates the token refuses
any field it has no descriptor for:

  InvalidArgument: request: Invalid value: unknown field with protobuf tag 10001

so every quiesce failed and the fenced Resume before a claim's release
was retried forever: the claim was never released and the session
refused every turn after its first.

The Substrate client now detects that refusal, remembers that its
ate-api takes no token, logs one warning naming the lost fence, and
sends that request and every later one without the token. The runtime
boundary works as before the token; only a frozen holder's late Pause or
Suspend is no longer refused. ate-api exposes no version a controller
could check at startup, and Substrate is usually installed on its own,
so the chart's dependency cannot hold the pair together.

Fixes #230

Signed-off-by: Timo Derstappen <teemow@gmail.com>
Signed-off-by: Timo Derstappen <teemow@gmail.com>
Co-authored-by: giantswarm-align-files[bot] <288255335+giantswarm-align-files[bot]@users.noreply.github.com>
Co-authored-by: Quentin Bisson <quentin.bisson@gmail.com>
Nothing wrote Harness.status, so kubectl get harness showed an empty READY
column and the gRPC Harness.ready was always false although the Harness
served.

The controller derives Ready from the Harness's WorkerPool and the golden
boots of the Agents that reference it, and writes it on its own status
queue. One successful golden boot proves the Harness's workload boots, so
it reads True; a failed boot reads False with its reason while no Agent
booted, since an Agent's own configuration can fail its boot too; Unknown
while nothing has tried the workload. The status write keeps
status.capabilities.

Refs kagent-dev#2764

Signed-off-by: Timo Derstappen <teemow@gmail.com>
Signed-off-by: Timo Derstappen <teemow@gmail.com>
Helm's and Flux's readiness wait reads a Ready condition other than True
as a rollout in progress, so a Harness without Agents, Ready Unknown,
held every upgrade of the chart that ships it until the wait timed out
and the release rolled back. Before any golden boot finished the
resolved WorkerPool keeps the Harness Ready (WorkerPoolResolved); a
successful boot reads Booted, a failed one with none booted False.

Signed-off-by: Timo Derstappen <teemow@gmail.com>
Signed-off-by: Timo Derstappen <teemow@gmail.com>
Signed-off-by: Timo Derstappen <teemow@gmail.com>
* fix(session): retry a failed quiesce once the Actor runs again

When the pause or suspend after a turn returned an error, settleIdleWork
re-read the Actor and released the claim without a snapshot as soon as it
was RUNNING again, for example after a first SuspendActor that outlived its
deadline and that Substrate then abandoned. The suspend was never retried,
so the turn kept no snapshot, and nothing on the session said why.

A RUNNING Actor now gets the pause or suspend once more, fenced with a
newer token, at the backoff settleIdleWork already uses; its outcome
settles the same way. The claim is released without a snapshot only when
the retry fails too, or the Actor is crashed or gone. The release records
why on the session as last_quiescence_failure (task, reason, message and
time), which GetSession, ListSessions and the CLI's session details show;
a later boundary with a snapshot clears it.

Signed-off-by: Timo Derstappen <teemow@gmail.com>

* docs(fork): carry the retry of a failed quiesce

Signed-off-by: Timo Derstappen <teemow@gmail.com>

---------

Signed-off-by: Timo Derstappen <teemow@gmail.com>
…ns (#227)

* ci(telemetry): fetch the contract's upstream sources once at their pins

The Telemetry Contract Check cloned three GitHub repositories on every
Weaver run: the two registries telemetry/registry/manifest.yaml depends
on (the GenAI registry re-cloning the core one through its own
manifest) and the shared policy packages, eleven clones per
`make semconv-verify`. A clone that flaked turned the check red on a
pull request's first run, and the result of the check depended on
reaching GitHub.

The three sources are now pinned in telemetry/versions.env and fetched
once by `make semconv-deps` into the git-ignored telemetry/deps: a
depth-one fetch of the pinned ref per repository, reused without
touching the network while its pin stands. The manifest names the
checkouts, semconv-check passes the policies from them, and the fetch
points a fetched registry's own git dependency at the checkout of the
same pin, refusing a pin that wants another ref, since the contract and
its dependencies must agree on the core conventions. CI caches the
directory keyed on the pins.

Weaver's `[resolve.schema_url_overrides]` does not serve here: in
v0.26.1 and v0.27.0 the loader of `registry check` and `registry
generate` clones a manifest dependency before any override is consulted;
the overrides serve the on-demand resolver of live-check and serve.

Proof: `make semconv-verify` with the checkouts present, the Weaver
container on `--network none` and a dead HTTPS proxy on the host, exits
0 after the check, the four policy fixtures, the four generations and
the drift check.

Signed-off-by: Timo Derstappen <teemow@gmail.com>

* docs(fork): ledger row for the telemetry contract's pinned sources

Signed-off-by: Timo Derstappen <teemow@gmail.com>

* ci(telemetry): fetch the pinned sources before generating, name the core pin apart

semconv-generate reads the manifest's registries from telemetry/deps, so it runs semconv-deps like semconv-check. The core registry's pin is SEMCONV_CORE_REGISTRY: the Makefile includes versions.env and names the kagent registry SEMCONV_REGISTRY.

Signed-off-by: QuentinBisson <quentin@giantswarm.io>

---------

Signed-off-by: Timo Derstappen <teemow@gmail.com>
Signed-off-by: QuentinBisson <quentin@giantswarm.io>
Co-authored-by: QuentinBisson <quentin@giantswarm.io>
Co-authored-by: Quentin Bisson <quentin.bisson@gmail.com>
…unner

TestServe polled GetProcess for the first process's COMPLETED within
require.Eventually's one second. On a loaded runner the shell's start
plus the poll take longer, so the test failed with "Condition never
satisfied" and passed on rerun. Name one bound of five seconds, like the
cleanup's stop; a passing run still returns at the first COMPLETED.

Signed-off-by: Timo Derstappen <teemow@gmail.com>
…esourceVersion (#245)

The Harness and ModelConfig status writes were an UpdateStatus of a copy of
the informer cache, so a write the cache had not observed yet failed them
with a Conflict. Both are now a JSON merge patch of the status subresource
that carries only the fields the controller derives; the Harness patch never
carries status.capabilities.

Signed-off-by: QuentinBisson <quentin@giantswarm.io>
TestSandboxTemplateCatalog and TestSandboxKRTInformerQueueAndRestartCleanup start an
envtest API server and fail a plain go test of their package when the envtest
binaries are not set up. Skip them with a message naming KUBEBUILDER_ASSETS, and
let make test set the variable through setup-envtest, as the CI job does.

Closes #232

Signed-off-by: Timo Derstappen <teemow@gmail.com>
…r-token propagation (#254)

Substrate 1.5's egress replaces a credential header only when the request
carries it. The Claude forwarder deleted every server's compiled
Authorization and set the caller's token in its place, so a Secret-backed
Authorization was never sent without a caller token and overwrote the
caller's token with one.

A server that configures Authorization keeps its gateway placeholder; any
other takes the caller's token, or none. The adapter resolves the
compiler's credential references before forwarding. CompileCredentials
refuses a caller-owned MCP server on a host with a gateway authorization
binding when KAGENT_PROPAGATE_TOKEN is on.

Closes #253
…252)

* fix(skills): git sends the Authorization the egress gateway replaces

Substrate's egress credential effect replaces the value of a header a
request carries and leaves a request without it untouched. Git sends no
Authorization before a challenge, so a private git skill was fetched
without its credential, answered 401 and the materializer failed on a
terminal prompt.

The compiled source of a git artifact with a credentialRef is marked
authenticated, and the materializer has git send a placeholder
Authorization header for that source's URL only, configured for the one
invocation and never written to a gitconfig. Git never prompts, so a
refused credential fails with its own error. A public source sends no
header.

Signed-off-by: Timo Derstappen <teemow@gmail.com>

* docs(api): credentialRef reaches only requests that carry the header

The egress gateway replaces an Authorization header a request carries and
adds none, so the field's description and the ledger row no longer claim
the credential goes on every request to the host, or to the host's public
sources.

Signed-off-by: QuentinBisson <quentin@giantswarm.io>

---------

Signed-off-by: Timo Derstappen <teemow@gmail.com>
Signed-off-by: QuentinBisson <quentin@giantswarm.io>
Co-authored-by: QuentinBisson <quentin@giantswarm.io>
@QuentinBisson
QuentinBisson force-pushed the feat/claude-harness-git-leg branch from b55bc99 to f168353 Compare October 8, 2026 21:30
…or every client kind (#256)

* test(e2e): prove the egress credential contract through the gateway for every client kind

Each client a Secret-backed binding covers (model keys, RemoteMCPServer
headersFrom on every harness and on Claude with KAGENT_PROPAGATE_TOKEN,
git skill and plugin sources with a credentialRef) talks to a recording
upstream through Substrate's egress gateway. The upstream must see the
Secret value on every request and the placeholder on none, so a client
that sends no header fails against a gateway that replaces only present
headers.

The git upstream is git http-backend on the test host, served over HTTPS
under a cluster Service name. The CI job creates a CA per run, has the
gateway trust it (atenetEgress.upstreamTrust.caBundle) and passes it to
the tests to sign that upstream. Golden boots fetch git sources in
atespace ate-golden, so the job lets it resolve kagent Secrets.

Signed-off-by: QuentinBisson <quentin@giantswarm.io>

* fix(translator): send an Ollama Cloud model of the Python runtime to api.ollama.com

The compiler left OLLAMA_API_BASE unset for a cloud model so that the
runtime would route it, but only the Go runtime routes: the Python
runtime sent a cloud model to localhost:11434. The compiler now writes
the cloud endpoint whenever OllamaReachesCloud routes there. The Ollama
SDK sends OLLAMA_API_KEY, the gateway placeholder, as the bearer, which
a unit test now pins.

Signed-off-by: QuentinBisson <quentin@giantswarm.io>

---------

Signed-off-by: QuentinBisson <quentin@giantswarm.io>
…ough them

KAGENT_CLAUDE_CALLER_ROUTES maps a host to an upstream that acts on the turn
caller's credential for that host. The forwarder serves each host on loopback
under /route/<host>/ only while a turn runs, and git reaches it through
GIT_CONFIG_* insteadOf, so git clone and push run as the caller with no token
in the sandbox.
…'s rewrite scope

serve reads the turn's credential once and hands it to the proxy with the
path suffix, so a route request admitted during a turn carries that turn's
credential and a request admitted between turns carries none, whatever Bind
or Clear does while it is in flight. A real-git test asserts that only
https://<host>/ of a routed host is rewritten and that the loopback token
matches the forwarder's origin only. The ledger row and README say the route
is clone and fetch on the platform, with push left to the upstream.

Signed-off-by: QuentinBisson <quentin@giantswarm.io>
…l is cleared

A request admitted during a turn kept the caller's credential after Clear and
went on acting as the caller, which a long git fetch or one started in the
background by a tool makes routine. Each forwarded request now ends with the
turn whose credential it carries.

Signed-off-by: QuentinBisson <quentin@giantswarm.io>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants