feat(eval): collect Codex token usage

docs/superpowers/plans/2026-07-08-codex-token-usage.md

@@ -0,0 +1,167 @@
+# Codex Token Usage Implementation Plan
+
+> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.
+
+**Goal:** Add per-run Codex CLI input/output token collection to the eval runner, CSV output, SQLite `runs` schema, and existing `TokenUsage` metric path.
+
+**Architecture:** The runner accepts `--codex-usage-jsonl`, parses a Codex CLI JSONL file into a small `CodexTokenUsage` value, and stores those totals on `RunRow`. CSV and SQLite schemas append `model_input_tokens` and `model_output_tokens`; the Python metric extractor already consumes those column names.
+
+**Tech Stack:** Go eval runner and evalrun SQLite writer, SQLite schema SQL, Python pytest metric tests.
+
+---
+
+### Task 1: Codex Usage Parser
+
+**Files:**
+- Create: `multi-agent/tools/eval/runner/codex_usage.go`
+- Test: `multi-agent/tools/eval/runner/codex_usage_test.go`
+
+- [ ] **Step 1: Write failing parser tests**
+
+Add tests that call `ParseCodexUsageJSONL(path)` on JSONL fixtures. Cover summed usage records, alias fields, wrapper objects, ignored non-usage records, malformed JSON, unreadable files, and negative token counts.
+
+- [ ] **Step 2: Run red test**
+
+Run: `go test ./tools/eval/runner -run 'TestParseCodexUsageJSONL'`
+
+Expected: FAIL because `ParseCodexUsageJSONL` and `CodexTokenUsage` do not exist.
+
+- [ ] **Step 3: Implement minimal parser**
+
+Implement `CodexTokenUsage`, `ParseCodexUsageJSONL`, and helper functions in `codex_usage.go`. Read line by line, unmarshal to `map[string]any`, find usage maps at top level and under `event`, `response`, and `message`, sum non-negative integer values, and return an error for malformed JSON or negative values.
+
+- [ ] **Step 4: Run green test**
+
+Run: `go test ./tools/eval/runner -run 'TestParseCodexUsageJSONL'`
+
+Expected: PASS.
+
+### Task 2: Runner CSV Columns
+
+**Files:**
+- Modify: `multi-agent/tools/eval/runner/writer.go`
+- Modify: `multi-agent/tools/eval/runner/writer_test.go`
+
+- [ ] **Step 1: Write failing CSV tests**
+
+Update frozen column tests to expect appended `model_input_tokens` and `model_output_tokens`. Add a row roundtrip assertion that a `RunRow` with token values serializes those values into the new columns.
+
+- [ ] **Step 2: Run red test**
+
+Run: `go test ./tools/eval/runner -run 'TestCSVColumns|TestRunRow_CodexTokenUsage_RoundTripCSV'`
+
+Expected: FAIL because the columns and fields do not exist.
+
+- [ ] **Step 3: Implement CSV fields**
+
+Add `ModelInputTokens int` and `ModelOutputTokens int` to `RunRow`, append column names to `CSVColumns`, and append values in `rowAsCSVRecord`.
+
+- [ ] **Step 4: Run green test**
+
+Run: `go test ./tools/eval/runner -run 'TestCSVColumns|TestRunRow_CodexTokenUsage_RoundTripCSV'`
+
+Expected: PASS.
+
+### Task 3: Runner Flag and Row Assembly
+
+**Files:**
+- Modify: `multi-agent/tools/eval/runner/main.go`
+- Modify: `multi-agent/tools/eval/runner/runner.go`
+- Test: `multi-agent/tools/eval/runner/runner_test.go`
+
+- [ ] **Step 1: Write failing runner test**
+
+Add a test that supplies `Opts.CodexUsageJSONL` with a JSONL fixture and asserts `res.Row.ModelInputTokens` and `res.Row.ModelOutputTokens` contain the parsed totals.
+
+- [ ] **Step 2: Run red test**
+
+Run: `go test ./tools/eval/runner -run 'TestRun_RecordsCodexUsageJSONL'`
+
+Expected: FAIL because `Opts.CodexUsageJSONL` is not wired.
+
+- [ ] **Step 3: Implement flag and row wiring**
+
+Add `CodexUsageJSONL string` to `Opts`, add `--codex-usage-jsonl` in `main.go`, parse usage in `Run` before row assembly, return exit code 2 on parse errors, and stamp totals into `RunRow`.
+
+- [ ] **Step 4: Run green test**
+
+Run: `go test ./tools/eval/runner -run 'TestRun_RecordsCodexUsageJSONL'`
+
+Expected: PASS.
+
+### Task 4: SQLite Runs Schema
+
+**Files:**
+- Modify: `multi-agent/internal/observerstore/schema.sql`
+- Modify: `multi-agent/internal/evalrun/schema.go`
+- Modify: `multi-agent/internal/evalrun/writer.go`
+- Modify: `multi-agent/internal/evalrun/writer_test.go`
+- Test: `multi-agent/internal/evalrun/schema_test.go`
+
+- [ ] **Step 1: Write failing DB tests**
+
+Update roundtrip tests to insert and read token counts. Update schema drift expectations to require the two appended columns and detect missing token columns.
+
+- [ ] **Step 2: Run red test**
+
+Run: `go test ./internal/evalrun`
+
+Expected: FAIL because schema, expected columns, insert SQL, and `Schema` do not include token fields.
+
+- [ ] **Step 3: Implement DB schema changes**
+
+Append the two integer columns to `schema.sql`, `Schema`, `expectedColumns`, and `insertSQL`. Pass `s.ModelInputTokens` and `s.ModelOutputTokens` to `ExecContext`.
+
+- [ ] **Step 4: Run green test**
+
+Run: `go test ./internal/evalrun`
+
+Expected: PASS.
+
+### Task 5: Metric Extraction Verification
+
+**Files:**
+- Modify: `multi-agent/tools/eval/metrics/tests/test_metrics_user_promoted.py`
+
+- [ ] **Step 1: Write failing metric test**
+
+Add a test that creates a temp DB from a fixture schema with `model_input_tokens` and `model_output_tokens`, inserts two selected runs, and asserts `TokenUsage` returns summed input/output plus count.
+
+- [ ] **Step 2: Run red test**
+
+Run: `python -m pytest tests/test_metrics_user_promoted.py`
+
+Expected: FAIL until the fixture creates the new columns or inserts valid rows with the new schema.
+
+- [ ] **Step 3: Make metric fixture use new columns**
+
+Adjust the test fixture setup so the DB has both token columns and the inserted rows have known values.
+
+- [ ] **Step 4: Run green test**
+
+Run: `python -m pytest tests/test_metrics_user_promoted.py`
+
+Expected: PASS.
+
+### Task 6: Final Verification
+
+**Files:**
+- All changed files above.
+
+- [ ] **Step 1: Run focused Go tests**
+
+Run: `go test ./tools/eval/runner ./internal/evalrun`
+
+Expected: PASS.
+
+- [ ] **Step 2: Run metric tests**
+
+Run from `multi-agent/tools/eval/metrics`: `python -m pytest tests/test_metrics_user_promoted.py`
+
+Expected: PASS.
+
+- [ ] **Step 3: Inspect worktree**
+
+Run: `git status --short`
+
+Expected: only intentional files changed.

docs/superpowers/specs/2026-07-08-codex-token-usage-design.md

@@ -0,0 +1,53 @@
+# Codex Token Usage Design
+
+## Goal
+
+Collect per-run Codex CLI input and output token counts in the evaluation runner, so the existing `TokenUsage` metric can report real values from `runs.model_input_tokens` and `runs.model_output_tokens`.
+
+## Scope
+
+This change supports Codex CLI only. It does not normalize usage from Claude, OpenHands, E2B, or other agents. If no Codex usage JSONL is supplied, token fields remain zero.
+
+## Interface
+
+Add a runner CLI flag and matching `Opts` field:
+
+```text
+--codex-usage-jsonl <path>
+```
+
+The file is parsed after the workload oracle runs and before `RunRow` assembly. The parser reads Codex CLI JSONL line by line, ignores lines without usage data, and sums usage records.
+
+Accepted usage keys:
+
+- input: `input_tokens`, `prompt_tokens`
+- output: `output_tokens`, `completion_tokens`
+
+The parser accepts `usage` objects at the top level and under common wrappers such as `event`, `response`, and `message`.
+
+## Persistence
+
+Append two numeric fields to runner CSV:
+
+- `model_input_tokens`
+- `model_output_tokens`
+
+Append two integer columns to `runs`:
+
+- `model_input_tokens INTEGER NOT NULL DEFAULT 0`
+- `model_output_tokens INTEGER NOT NULL DEFAULT 0`
+
+The existing Python metric extractor already probes for these two columns and sums them, so no metric API change is needed.
+
+## Errors
+
+An unreadable or malformed supplied JSONL file is a preflight-style runner error with exit code 2. Individual JSONL lines without usage are ignored. Negative token counts are rejected.
+
+## Testing
+
+Use TDD:
+
+1. Add parser tests for Codex JSONL usage summing and ignored non-usage lines.
+2. Add runner/CSV tests for appended columns and row serialization.
+3. Add evalrun DB roundtrip and schema drift tests for the new columns.
+4. Add a Python metric test showing `TokenUsage` returns real sums when the columns exist.

multi-agent/cmd/evalrun-export/main.go

@@ -19,7 +19,9 @@ import (
 	_ "modernc.org/sqlite"
 )
 
-// columnNames is the 24-column header in the order defined by
+const runsColumnCount = 26
+
+// columnNames is the 26-column header in the order defined by
 // docs/specs/wt1-run-schema.spec.md §3. The CLI is intentionally
 // dependency-free with respect to internal/evalrun beyond this list —
 // it does NOT call into the Writer; it just reads the runs table.
@@ -48,6 +50,8 @@ var columnNames = []string{
 	"artifact_hashes",
 	"observer_trace_path",
 	"model_trace_id",
+	"model_input_tokens",
+	"model_output_tokens",
 }
 
 // usageError is exit code 2 — bad arguments. runtimeError is exit 1.
@@ -183,7 +187,7 @@ const (
 	selectRunsFilterSQL = `SELECT ` + columnSelectList + ` FROM runs WHERE experiment_id = ? ORDER BY run_id`
 )
 
-// columnSelectList is the 24-column projection in the canonical order.
+// columnSelectList is the 26-column projection in the canonical order.
 // Constant so the SELECT statements above are pure literals; refactors
 // that touch column order must update this AND the Insert path AND
 // schema.sql in lock-step.
@@ -193,7 +197,8 @@ machine_topology, context_ground_truth, capability_snapshot_hash,
 task_contract_hash, dynamic_mcp_registry_hash, selected_context,
 ground_truth_context, start_time, end_time, success_oracle_result,
 failure_category, human_intervention_count, artifact_hashes,
-observer_trace_path, model_trace_id`
+observer_trace_path, model_trace_id, model_input_tokens,
+model_output_tokens`
 
 // queryRuns runs the appropriate SELECT and returns *sql.Rows.
 func queryRuns(ctx context.Context, db *sql.DB, filterExperiment string) (*sql.Rows, error) {
@@ -203,44 +208,49 @@ func queryRuns(ctx context.Context, db *sql.DB, filterExperiment string) (*sql.R
 	return db.QueryContext(ctx, selectRunsFilterSQL, filterExperiment)
 }
 
-// humanCountIdx is the canonical position of human_intervention_count
-// in the 24-column ordering. The CSV path treats this column like the
-// others (string round-trip via strconv); the JSONL path emits it as
-// a JSON number.
-const humanCountIdx = 20
+// Canonical positions of INTEGER columns. The CSV path treats these
+// columns like the others (string round-trip via strconv); the JSONL
+// path emits them as JSON numbers.
+const (
+	humanCountIdx        = 20
+	modelInputTokensIdx  = 24
+	modelOutputTokensIdx = 25
+)
 
-// runRow holds one scanned row as 24 string cells. human_intervention_count
-// is the only non-string column; it's converted to its decimal string
-// form in scanOneRow so CSV/JSONL writers can decide whether to quote
-// or emit as a number.
+// runRow holds one scanned row as string cells. INTEGER columns are
+// converted to decimal string form in scanOneRow so CSV/JSONL writers
+// can decide whether to quote or emit as numbers.
 type runRow struct {
-	values [24]string
+	values [runsColumnCount]string
 }
 
 func scanOneRow(rows *sql.Rows) (*runRow, error) {
 	var (
 		r runRow
-		// int64 (not int) so a row written with
-		// human_intervention_count > 2^31 on a 32-bit build doesn't
-		// silently truncate to a negative number. SQLite INTEGER is
-		// up to 8 bytes signed; the column NOT NULL DEFAULT 0 means
-		// Scan never sees NULL — drift guard catches the NULL-able
-		// drift variant separately.
-		humanCount int64
-		// 24 dest pointers in the canonical order.
-		dest [24]any
+		// int64 (not int) so a row written with an SQLite INTEGER
+		// > 2^31 on a 32-bit build doesn't silently truncate.
+		humanCount, modelInputTokens, modelOutputTokens int64
+		// dest pointers in the canonical order.
+		dest [runsColumnCount]any
 	)
-	for i := 0; i < 24; i++ {
-		if i == humanCountIdx {
+	for i := 0; i < runsColumnCount; i++ {
+		switch i {
+		case humanCountIdx:
 			dest[i] = &humanCount
-		} else {
+		case modelInputTokensIdx:
+			dest[i] = &modelInputTokens
+		case modelOutputTokensIdx:
+			dest[i] = &modelOutputTokens
+		default:
 			dest[i] = &r.values[i]
 		}
 	}
 	if err := rows.Scan(dest[:]...); err != nil {
 		return nil, err
 	}
 	r.values[humanCountIdx] = strconv.FormatInt(humanCount, 10)
+	r.values[modelInputTokensIdx] = strconv.FormatInt(modelInputTokens, 10)
+	r.values[modelOutputTokensIdx] = strconv.FormatInt(modelOutputTokens, 10)
 	return &r, nil
 }
 
@@ -261,7 +271,7 @@ func exportCSV(ctx context.Context, db *sql.DB, w io.Writer, filterExperiment st
 		if err != nil {
 			return fmt.Errorf("scan row: %w", err)
 		}
-		escaped := make([]string, 24)
+		escaped := make([]string, runsColumnCount)
 		for i, v := range r.values {
 			escaped[i] = csvEscape(v)
 		}
@@ -295,8 +305,8 @@ func csvEscape(s string) string {
 }
 
 // exportJSONL writes one JSON object per line. Key order matches
-// columnNames; human_intervention_count is emitted as a number, all
-// other fields as strings.
+// columnNames; INTEGER columns are emitted as numbers, all other
+// fields as strings.
 func exportJSONL(ctx context.Context, db *sql.DB, w io.Writer, filterExperiment string) error {
 	rows, err := queryRuns(ctx, db, filterExperiment)
 	if err != nil {
@@ -324,9 +334,9 @@ func exportJSONL(ctx context.Context, db *sql.DB, w io.Writer, filterExperiment
 }
 
 // orderedJSONLine custom-marshals one row preserving columnNames order
-// and emitting human_intervention_count as a JSON number.
+// and emitting INTEGER columns as JSON numbers.
 type orderedJSONLine struct {
-	values [24]string
+	values [runsColumnCount]string
 }
 
 func (o orderedJSONLine) MarshalJSON() ([]byte, error) {
@@ -344,13 +354,10 @@ func (o orderedJSONLine) MarshalJSON() ([]byte, error) {
 		buf = append(buf, keyJSON...)
 		buf = append(buf, ':')
 		// value
-		if i == humanCountIdx {
-			// human_intervention_count: numeric. scanOneRow built this
-			// via strconv.FormatInt(_, 10) on an int64, so it is always
-			// a JSON-grammar-conformant decimal integer (sign + digits,
-			// no whitespace, no exponent). Direct append into the JSON
-			// stream is therefore safe — no escaping or re-encoding
-			// pass needed.
+		if isJSONNumberColumn(i) {
+			// Numeric columns are built via strconv.FormatInt(_, 10)
+			// on int64, so they are always JSON-grammar-conformant
+			// decimal integers. Direct append is safe.
 			buf = append(buf, []byte(o.values[i])...)
 		} else {
 			vJSON, err := json.Marshal(o.values[i])
@@ -363,3 +370,12 @@ func (o orderedJSONLine) MarshalJSON() ([]byte, error) {
 	buf = append(buf, '}')
 	return buf, nil
 }
+
+func isJSONNumberColumn(i int) bool {
+	switch i {
+	case humanCountIdx, modelInputTokensIdx, modelOutputTokensIdx:
+		return true
+	default:
+		return false
+	}
+}