feat: enable + verify async/KEDA leaf-worker path on OCP (dynamic queue-driven scaling)

[cwiklik]

Summary

Enable and verify the async leaf-execution path (POST /runs {async:true} → Redis Streams work-queue → KEDA-scaled leaf-worker Jobs) end-to-end on OCP, so the harness scales dynamically and asynchronously on backlog instead of relying on the manual pre-scale workaround.

This is a finish + verify, not a build — the machinery is ~95% present.

Why

Long agentic leaves (8–16 min), bursty, idle-heavy. The synchronous /runs path holds one HTTP connection per leaf for its whole duration → head-of-line blocking, ~45% transport loss, and it defeats Knative autoscaling (the harness's own 503 {saturated} sheds excess faster than the KPA reacts, so pods stay at ~1). We've been papering over this with a manual pre-scale hack in the experiment drivers.

The async path fixes all of it: KEDA (not Knative) scales workers by watching the Redis stream backlog; the Knative harness becomes a thin 202 acceptor → no long-held connections, no 503/pre-scale problem, transport loss gone.

What already exists (main @ #153)

• Redis Streams work-queue — stream leaf-queue, group leaf-workers; XADD/XAUTOCLAIM/XREADGROUP/XACK, heartbeat, dead-letter reap, idle-consumer GC (packages/work-queue/src/queue.ts).
• Async route — POST /runs {async:true} → enqueue → 202 accepted+sessionId; GET /runs/status?sessionId= reads from Redis (packages/knative-server/src/server.ts).
• Worker — claim-one → run → write result → ack, with retry classification (leaf-job.ts, harness/src/leaf-job-runner.ts, classify-outcome.ts).
• KEDA ScaledJob — redis-streams scaler on leaf-queue/leaf-workers, dual triggers lagCount (new work + scale-to-zero) + pendingEntriesCount (reclaimer), pollingInterval 5s, maxReplicaCount 10, run-once Jobs (deploy/knative/leaf-scaledjob.yaml).
• --with-keda installs KEDA (RH Custom Metrics Autoscaler) + KedaController + waits for the CRD (deploy/knative/setup-ocp.sh).
• Pool leasing — least-loaded + Redis lease + saturation error (harness/src/select-sandbox.ts).
• E2E smoke — deploy/knative/leaf-async-smoke.sh.

Remaining work (the actual scope)

• ScaledJob → pool leasing. leaf-scaledjob.yaml pins KAGENTI_SANDBOX_NAME=sandbox-0 and sets no KAGENTI_SANDBOX_POOL_SELECTOR → all 10 workers would collide on one sandbox. Set the pool selector (+ cap/ttl) so workers lease from the pool.
• ScaledJob image. Template is dev.local/serverless-harness:local; wire it into the overlays/ocp image transform (else it applies raw dev.local → ImagePullBackOff, same gotcha as the sandbox-relay).
• Wire the apply. Confirm/complete that --with-keda actually applies leaf-scaledjob.yaml (+ the serverless-harness SA/RBAC the Job needs) after installing KEDA.
• Driver → async. Switch the E1/E6 drivers from the sync /runs (hold-connection) to POST {async:true} → poll /runs/status (lib.sh already has poll_leaf_result). Also eliminates the ~45% transport loss and retires the pre-scale hack.
• Tune maxReplicaCount vs pool × lease-cap. Excess workers correctly hit SandboxPoolSaturatedError → requeue (clean back-pressure); align numbers intentionally (pool × cap ≈ target concurrency).
• OCP end-to-end verify. setup-ocp --with-keda → leaf-async-smoke.sh → burst-enqueue → watch KEDA spawn Jobs → drain → scale-to-zero; confirm paused/gated leaves + the reclaimer.

Acceptance criteria

• Enqueue a burst of N leaves → KEDA scales leaf-worker Jobs on leaf-queue lag (up to maxReplicaCount) and scales to 0 when drained.
• Workers lease from the sandbox pool (no sandbox-0 collision); over-provisioned workers back-pressure via requeue, not failure.
• ScaledJob runs the real image (not dev.local).
• E1/E6 drivers run via the async path with no pre-scale and no transport loss.
• Crashed-worker entries are reclaimed (dead-letter path) within the configured idle window.

Relationship to other work

• Retires the uncommitted pre-scale fix (currently stranded in the durable clone; keep as interim until this is verified).
• Complements feat: multi-protocol model provider (Anthropic + OpenAI-compat) — support LiteLLM and RITS from main #157 (multi-protocol model provider) — orthogonal; both feed the same experiment harness.

Assisted-By: Claude Code