functional-tests: every triage case fails with API Error policy_denied on main since 2026-09-03

[waynesun09]

What happened

functional-tests fails on main itself: every triage case gets API Error: Error code policy_denied on the very first Vertex call (agent runs ~2 s, apiKeySource: none, $0.00), then validation fails after 2 iterations.

• Control: workflow_dispatch on main (2a10349) — https://github.com/fullsend-ai/fullsend/actions/runs/33763731846 — 4/4 FAIL.
• Last run that actually executed the step and passed: https://github.com/fullsend-ai/fullsend/actions/runs/33704432608 (push to main b498a8c, 2026-09-03 01:38Z).
• Every "green" functional run on PRs since then skipped the step (Check for functional-test-relevant changes → no relevant files), e.g. 33756644901, 33753912927, 33751315497, so the breakage was invisible until a PR touched internal/.
• First visible failure: PR feat(runtime): add the steer interface, off by default #6959, runs 33758001770 (both attempts) and three workflow_dispatch bisect runs on that branch; all identical.

Case output (from the eval-results artifact): → Agent: claude-opus-5 (v2.1.258), two api_retry events with error: unknown, then API Error: Error code policy_denied. One case in one attempt also hit an OpenShell provider race (default credential was modified concurrently during persist credential) with 4 parallel cases.

Suspects in the window (01:43Z → 12:55Z)

• chore(harness): repin sandbox and code images to the 0.40.0 release digests agents#1160 (11:19Z): repin of the sandbox and code images to the 0.40.0 release digests (fullsend-sandbox@sha256:f8255971…, claude 2.1.258). The functional job resolves fullsend-ai/agents@main, so it picked this up immediately.
• fullsend feat(#6649)!: add packages read permission to coder and fix roles #6654 (01:43Z) is mint/appsetup only and the functional run mints nothing (No --mint-url provided), so it is unlikely.

Not verified: whether the denied request is the Vertex model call itself or an earlier call the new claude version makes; the gateway's denial log on the runner is not in the artifact.

Validation criteria

workflow_dispatch of functional-tests.yml on main executes the four triage cases and passes.

[fullsend-ai-triage]

🤖 Finished Triage · ✅ Success · Started 1:59 PM UTC · Completed 2:04 PM UTC

Commit: 2a10349 · View workflow run →

Runtime: claude · Model: opus → claude-opus-4-6 · Effort: high · Cost: $0.84

[fullsend-ai-triage]

Triage Summary

Severity: Critical — all triage functional tests are broken on main, and the breakage was invisible for ~11 hours due to path-filter skipping on PR runs.

Root cause hypothesis

The most likely cause is fullsend-ai/agents#1160 (merged 11:40Z), which repinned harness images from 0.37.0 to 0.40.0 (fullsend-sandbox@sha256:f8255971, claude v2.1.258). The functional-tests workflow resolves fullsend-ai/agents@main at runtime, so it immediately picked up the new images. The policy_denied error on the first Vertex call suggests the new claude version or sandbox image makes API requests that violate a Model Armor policy rule — either the model call format changed or the new version makes an additional call the policy doesn't allow.

The concurrent credential race (default credential was modified concurrently during persist credential) observed in one case under 4-way parallelism may be a separate issue in the 0.40.0 sandbox's credential handling.

Recommended investigation

1. Confirm the suspect: Run functional-tests with agents pinned to pre-mint: add retro and prioritize roles to test defaults #1160 (0.37.0 images). If tests pass, the repin is the cause.
2. Check Vertex gateway logs for the specific Model Armor policy rule being triggered.
3. Short-term mitigation: Revert agents image pins to 0.37.0 while the policy/image issue is resolved.
4. Investigate credential race under parallel case execution separately.

Proposed test case

Run functional-tests workflow_dispatch on main:
- All 4 triage cases should pass
- Each case should show non-zero cost (real Vertex API usage)
- No policy_denied or api_retry errors in eval-results
- No concurrent credential modification errors

Related issues

• Harness should fail fast on authentication errors instead of retrying #2058 — Harness should fail fast on authentication errors instead of retrying (would improve signal for errors like this)
• Functional tests use stale agents v0 tag, miss post-triage fixes #4931 — Functional tests use stale agents tag (related to agents version resolution)

---

Labels: CI pipeline bug affecting triage agent functional tests with critical severity — all tests broken on main.

---

Next steps:

• /fs-code — agent creates a PR to implement this issue
• /fs-code <your instruction> — agent implements with your specific guidance

[fullsend-ai-coder]

🤖 Finished Code · ❌ Failure (post-script /home/runner/work/fullsend/fullsend/.fullsend/.fullsend-cache/resources/sha256/21644e0238d0b83db763fa138c412322223b0ba4a919a542b839e11b6a3d62e7/scripts/post-code.sh failed: exit status 1) · Started 2:06 PM UTC · Completed 2:31 PM UTC

Commit: 2a10349 · View workflow run →

Runtime: claude · Model: opus → claude-opus-4-6 · Effort: high · Cost: $10.15

[fullsend-ai-coder]

⚠️ Post-code script failed — Push rejected — workflows permission (exit code 1)

The code agent completed, but the post-code script failed before finishing.

Security boundary: the coder app intentionally lacks workflows write permission. Changes to .github/workflows/ must be made outside the agent (e.g., via a manual PR). Re-run the agent without workflow file changes, or apply those changes separately.

Workflow run: https://github.com/fullsend-ai/fullsend/actions/runs/33764725541

Details:
To https://github.com/fullsend-ai/fullsend.git
! [remote rejected] agent/6962-pin-functional-test-model -> agent/6962-pin-functional-test-model (refusing to allow a GitHub App to create or update workflow .github/workflows/functional-tests.yml without workflows permission)
error: failed to push some refs to 'https://github.com/fullsend-ai/fullsend.git'
To https://github.com/fullsend-ai/fullsend.git
! [remote rejected] agent/6962-pin-functional-test-model -> agent/6962-pin-functional-test-model (refusing to allow a GitHub App to create or update workflow .github/workflows/functional-tests.yml without workflows permission)
error: failed to push some refs to 'https://github.com/fullsend-ai/fullsend.git'
Please check the workflow logs for full details and retry with /fs-code if appropriate.

[waynesun09]

Root cause correction: this is not Model Armor and not model authorization. The gateway log in the eval-results artifact of run 33763731846 shows OpenShell's OPA denying the Claude Code binary itself:

NET:OPEN DENIED /usr/lib/node_modules/@anthropic-ai/claude-code/bin/claude.exe -> sts.googleapis.com:443
reason: binary '.../claude.exe' not allowed in policy '_provider_vertex_ai'

Claude Code 2.1.2xx is a Bun binary named claude.exe; the 0.40.0 image (2.1.258) runs it, and internal/scaffold/fullsend-repo/profiles/fullsend-vertex-ai.yaml only allowlists **/claude. The fixed profile from fullsend-ai/agents#1118 is fetched by the functional run, but <fullsend-dir>/profiles/ is imported afterwards and silently replaces it (FULLSEND_DIR=internal/scaffold/fullsend-repo). Fix tracked in #6971; pinning the model would not have helped.