feat: earn the moat — codex finalText receipt + honest copy + real-steering guard#161
Merged
Conversation
…nt_message codex has no single authoritative final-result string (claude uses the SDK m.result); its LAST assistant agent_message IS the answer. runCodex now stitches that onto run.completed as finalText (via attachCodexFinalText), so the truthful memory receipt (speechBubble), the answer caption, and the 'job done' notification light on the DEFAULT codex path — previously structurally dark (finalText only ever set on the claude path). Within-contract: finalText is already an optional field on the run.completed ActionEvent (shared/events.ts) — no union kind added, mapCodexThreadEvent stays PURE (the stitch is stateful-loop-only), no core/electron boundary crossing. LOCKSTEP (the guardrail): the deterministic proposal-eval fixture (proposalFixtures.ts) replays the PURE mapper, so it would NOT auto-reflect the loop stitch — it now replays the shared stitchCodexFinalText so the CI eval stays representative of production. Its test + the stale 'codex emits none' comment are reconciled in the same commit. Reproduce-first: stitchFinalText.test.ts pins last-message-wins, empty→suppressed, never-clobber-existing, pass-through — RED before the helpers existed. tsc clean; stitch + fixture + pure-mapper tests green (25). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Vsq543S6voPTUsGsEqZvnD
…checked', not 'used')
Once codex sets finalText (prior commit), the receipt fires on the DEFAULT codex path.
But the gate proves only that memory was RECALLED this run (a same-run recall status +
run.completed.finalText) — NOT that it shaped the answer. A true application gate is not
observable from the ActionEvent stream (would need provenance threaded through
decide→executor→event, an invasive frozen-union change). So the honest, renderer-local fix
is the COPY: 'used what I remember about your setup' → 'checked what I remember about your
setup'. Real steering is proven separately (verify:memory-steered-real / Phase 0: 5/5).
Adds a test for the REAL codex event order (message → run.completed{finalText:same}): the
bare message earns no receipt; the settled run.completed carries the answer + the receipt.
Existing receipt cases stay green (they assert against the exported MEMORY_RECEIPT_TEXT
constant, not a literal).
FOUNDER-TUNABLE: the exact copy is a taste call — flagged for review.
33 speechBubble tests green; tsc clean.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Vsq543S6voPTUsGsEqZvnD
…uard Turns the validated in-process experiment into a permanent guard: real qwen2.5:3b decide + real codex executor + a fixed-cipher disk store, measuring recall→args.task→actual-diff, treatment vs control. Renamed to src/core/memory2/memorySteered.live.test.ts, gated on RORO_STEERING_LIVE=1 (collect-but-skip in CI, like crosslaunch.live), run via 'npm run verify:memory-steered-real'. THE TEETH (the guardrail): it now ASSERTS the ROADMAP Arc A go/no-go — treatment steers >=60% on BOTH args.task and the real file diff, control 0 (no false positives) — instead of only console.log. First measured run: 5/5 · 5/5 · 0/5 · 0/5. Without the expect()s an opt-in test is vacuous green, worse than the deterministic fake-smoke it complements. KEEP the faked scripts/smoke-memory-steered.mjs (re-scoped its header): it uniquely covers packaged-app IPC/preload/orchestrator-over-CDP plumbing + the trace-capture path. The two are complementary — plumbing-determinism (fake models, CI-safe) vs real-model semantics (opt-in). Closes the ROADMAP 'real executor honoring the preference in a diff is a SEPARATE unproven check' gap; notes it there. tsc clean · 1096 passed / 12 skipped · eslint 0 errors. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Vsq543S6voPTUsGsEqZvnD
…ncy, doc drift Multi-hat review (correctness · contract-integrity · truthfulness · test-rigor · ripple): no blockers; both re-run hats (contract, ripple) PASS — finalText stays within the frozen optional field, mapper stays pure, boundary holds, all 7 finalText consumers correct (the codex double-emit is exact parity with claude). Fixes: - test-rigor: the runCodex STREAMING stitch (the production path) had no deterministic CI coverage — add runCodexStitch.test.ts (mocked spawn replays the captured fixture through the real loop, asserts run.completed.finalText). And the 'empty message' test placed the blank FIRST so it did not pin the non-empty guard — add a TRAILING-blank case that fails if the guard is removed (a trailing blank would silently wipe the answer). - truthfulness consistency: the OTHER receipt surface (turnReceipt 'Memory used.') still over-claimed — soften to 'Memory recalled.' so BOTH receipt surfaces claim recall, not application. Two stale 'used what I remember' code docstrings (speechBubble, bootstrap) and three docs (INTERACTION, floating-redesign, VERIFICATION) synced to 'checked'/'recalled'. - codex.ts: 'IS the answer' softened to best-effort-heuristic wording (trailing-non-answer edge). - contract nit: drop the dead attachCodexFinalText re-export from executor/index.ts. - guard: memorySteered.live asserts N>=1 (RORO_STEERING_N=0 would make the threshold vacuous). tsc clean · 1099 passed / 12 skipped · eslint 0 errors. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Vsq543S6voPTUsGsEqZvnD
…RED (codex review) Codex review caught a truthfulness gap in the guard: the seeded convention is 'every .py file BEGINS with the marker as its first line', but markerInFiles only checked whether the marker appeared ANYWHERE in ANY file — which could pass on a README while hello.py violates the rule, overstating what 'honored the diff' proves. Added markerAsPyHeader() (marker is the FIRST non-blank line of a created .py file) and moved the treatment assertion onto it, so the guard proves the executor actually HONORED the recalled convention. Control still asserts the marker appears nowhere. ROADMAP note reconciled (loose anywhere-run vs the strict per-header guard). No CI impact (opt-in/non-CI, collect-but-skip). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Vsq543S6voPTUsGsEqZvnD
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 440fac57dc
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
Comment on lines
+69
to
+72
| else if (name.endsWith('.py')) { | ||
| try { | ||
| const firstLine = readFileSync(p, 'utf8').split('\n').find((l) => l.trim().length > 0) ?? ''; | ||
| if (firstLine.includes(MARKER)) honored = true; |
There was a problem hiding this comment.
When the real Codex run creates more than one .py file, this helper returns true as soon as any one of them has the marker as its header, so the live steering guard can pass even if another generated Python file violates the seeded rule that every .py file must begin with the marker. That weakens the opt-in proof from “convention honored” to “at least one file honored it”; consider failing on the first nonmatching Python header and only passing when all generated .py files comply.
Useful? React with 👍 / 👎.
…iew round 2) Codex re-review sharpened the guard once more: checking the first line INCLUDES the marker on ANY .py file still over-claimed — a Python assignment line (MARKER = '...') or a compliant helper beside a non-compliant hello.py would pass. conventionHonored() now targets hello.py (the file the task creates) and asserts its first non-blank line EQUALS the exact header '# codename: <MARKER>'. This makes markerHonored prove exactly the seeded convention. Opt-in/ non-CI; no CI impact. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Vsq543S6voPTUsGsEqZvnD
…x review round 3) Codex round 3: (P1) markerHonored counted even if codex later run.failed/aborted — now gated on exec.ok (the run.completed ok), per the ROADMAP proof contract. (P2) the recursive hello.py search could match a nested/second compliant file masking a non-compliant root hello.py — conventionHonored now checks the repo-ROOT hello.py directly (codex runs -C repo), exact-header only. Opt-in/non-CI. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Vsq543S6voPTUsGsEqZvnD
…und 4) Codex round 4: skipping leading blanks + trimming let a hello.py starting with a blank line or leading spaces false-pass, looser than 'begin with the exact first line'. Now checks the literal line 0 (only a trailing CR stripped) == the exact header. Maximally faithful to the seeded convention. Opt-in/non-CI. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Vsq543S6voPTUsGsEqZvnD
…codex review round 5) Round 5: (1) the docstring still said 'first non-blank line' after the impl moved to literal line 0 — synced. (2) the 'real qwen2.5:3b' claim was not pinned (brain.decide honors an inherited OLLAMA_MODEL), so verify:memory-steered-real now pins OLLAMA_MODEL=qwen2.5:3b + OLLAMA_EMBED_MODEL=nomic-embed-text so the proof runs the model it claims. Opt-in/non-CI. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Vsq543S6voPTUsGsEqZvnD
gorajing
added a commit
that referenced
this pull request
Jul 6, 2026
…gue fixtures, docs absorb the first dogfooding session (#162) * fix(eval): repair the eval runners' post-W7 artifact paths + pin the path family in CI Since the W7 core-split (#149) moved src/brain -> src/core/brain, both live runners still wrote/read their artifacts under the stale 'src/brain/eval' (runEval.ts latest/baseline writes; runProposalEval.ts baseline read + proposalLatest/proposalBaseline writes) and crashed ENOENT at the end of a full run. The directory now lives once in eval/paths.ts (EVAL_DIR + evalArtifactPath), both runners use it, and paths.test.ts pins that EVAL_DIR resolves to the directory the eval files actually live in (plus that the checked-in baseline.json exists there) — sabotage-verified: reverting EVAL_DIR to the stale path fails both tests. Also moves the two stale .gitignore scratch entries (latest.json / proposalLatest.json) to the real output location so runner scratch stays untracked. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat(eval): make deflect-on-vague measurable — 5 vague-initiative DECIDE fixtures (fixture-first) First founder dogfooding session (2026-07-05): 'create something interesting' made the 3B answer 'Let's brainstorm a fun project idea together' — a chat deflection that fails ROADMAP §6's lands-or-clarifies bar (do the task or ask a PERTINENT question). This adds the failure class to the eval WITHOUT touching decide (measure first, fix later): - DecideCase grows alsoAccept (a golden SET for lands-or-clarifies cases) + a 'vague-initiative' tag (the DECIDE-side analogue of BehavioralTaxonomy). - 5 fixtures (vague-interesting/-cool/-fun/-buildme/-impress) accept exactly {run_agent, clarify}; answer FAILS. Expected to fail against the current model — that is the point. - scoreDecision gains the optional alsoAccept set (red-first unit tests: a deflection still scores wrong_command). - fixtures.test.ts pins the class deterministically in CI: acceptance set is exactly {run_agent, clarify}, and every vague case must NOT be caught by the deterministic clarify gate (same loud-re-label discipline as the marker-less pin), so the fixtures keep measuring the MODEL. Live tier stays opt-in/report-only; the deterministic suites stay green. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * chore(eval): refresh the shape-stale baseline via --write-baseline (real Ollama, new fixtures included) OLLAMA_AVAILABLE=1 npm run eval:brain -- --write-baseline against local qwen2.5:3b — the sanctioned baseline workflow; baseline.json is never hand-edited. The old snapshot predated the W3 5→50 fixture expansion and the new runner output shape. Old → new headline numbers: - decide: 22/22 (100%) → 26/27 (96.3%; one wrong_command: run-writetest → answer — DECIDE streams at temperature 0.3, a run is a snapshot) - extract: 10/10 (100%) → 10/10 (100%), unchanged - behavioral value-quality: 2/5 (40%, the original 5-case set) → 26/35 (74.3%, the full expanded set — reproduces the #145 report-only number) - NEW keys now captured: behavioralNull 7/15 (46.7%; all 8 misses are hard-negative false_facts) + behavioralByTaxonomy (hard-negative 20% is the weak family; multi-fact/supersede/marker-less 100%) MEASURED TRUTH on the new vague-initiative cases: contrary to expectation, the current 3B lands-or-clarifies on all 5 (and 30/30 in a 3×-repeat probe with and without a memory section) — the fixtures therefore PIN the class against regression; the live 2026-07-05 deflection has not reproduced from paraphrased transcripts, so the fixture-block comment now records that and points the follow-up at promoting a RORO_TRACE capture of a real deflecting turn. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * docs: absorb the dogfooding + floating/moat signal — LESSONS, HANDOFF, ROADMAP, VERIFICATION - LESSONS.md: new Dogfooding entry (2026-07-05) naming the four failure classes — deflect-on-vague / feedback-fade / working-illegibility / env-overrides-config — each with its file:line locus and fix status (2–4: the sibling dogfood-blocker slice; deflect-on-vague: fixture-first, with the measured 2026-07-06 twist that the paraphrased fixtures currently PASS, so the real repro needs a RORO_TRACE capture). - HANDOFF.md: §3 work log extended past W8 — the floating redesign (#153–#160, resting purity + truthful receipt), earn-the-moat (#161, codex finalText stitch + verify:memory-steered-real 5/5 vs 0/5), the first dogfooding session, and this signal-reconcile slice; §4 gains the refreshed-baseline numbers (the old bullets stay as the historical snapshot); §8 gains the corrected forward order (dogfood fixes → signing → legible-moment → executor-facts pilot → self-rehearsal → demo → stranger session). - docs/ROADMAP.md: Arc A §6 step 5 (founder dogfooding) marked STARTED 2026-07-05 with the four findings + the fixture-covered status of deflect-on-vague. - docs/VERIFICATION.md: the stale pre-W7 layout references in the cohort trace-review section (lines ~10–27) now point at src/core/… (the vitest command there was unrunnable). Stale paths in LATER sections of the file remain — out of this slice's scope, flagged in the handoff report. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(review): address the review panel — doc truthfulness + pin hardening (S-A round 1) Panel findings (all CONFIRMED, 0 false positives across the 5-hat review): - BLOCKER: four docs-of-record sites claimed feedback-fade was 'being fixed in the sibling dogfood-blocker slice' — it is NOT (it belongs to the upcoming legible-moment slice). LESSONS, HANDOFF x2, ROADMAP now say so honestly, and working-illegibility is split truthfully (sleep-mid-run half → dogfood-blockers; the distinct working pose → legible-moment). - LESSONS locus corrected: summoning the ask also pokes the presence clock, not only petting. - alsoAccept untag loophole closed: any DECIDE case widening its acceptance set must declare a failure-class tag (pinned in fixtures.test.ts) — an untagged alsoAccept was silent softening. - evalArtifactPath now __dirname-based (correct from ANY cwd, travels with the dir on a move); new source-grep pin: the runners must route through evalArtifactPath, no quoted hand-spelled eval dirs (the exact post-W7 breakage class), header-comment convention exempted. - Stale src/brain path sweep: cohortTraceReview.ts fixture pointer, COHORT_TRACE_TO_EVAL.md verify command, RUN.md:20, README.md x2, WS1 doc x2, ROADMAP live locus. tsc clean · 1104 passed / 12 skipped · eslint 0 errors. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Vsq543S6voPTUsGsEqZvnD * docs: sweep stale PGlite/pgvector memory claims + the temp-0 eval claim (codex S-A round 1) Codex caught materially-false user-facing docs surviving the reconcile: README (6 sites) and RUN.md (4 sites) still described memory as an in-process PGlite+pgvector database at src/memory2 — deleted in W5 (#147); memory is encrypted files-as-truth + a rebuilt-on-launch in-memory vector index at src/core/memory2, embeddings via local nomic-embed-text. Also HANDOFF's eval header said 'temp 0' while decide actually streams at temperature 0.3 (brain/index.ts:190) — the same header the new baseline note's temp-0.3 explanation contradicted. And the crosslaunch proof pointer moved to src/core/orchestrator/. All claims now match the code. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Vsq543S6voPTUsGsEqZvnD * docs: complete the pre-core-split path sweep + fix dangerous embedder advice (codex S-A round 2) - RUN.md's embedder note was DANGEROUS post-W5: it described a fixed vector(N) DB column and told users to move/delete the memory dir when changing embedders — that dir now holds the durable encrypted memories (files-as-truth); vectors.jsonl is the zero-authority derived cache that re-embeds automatically. Rewrote to the true architecture. - README executor pointers (table + tree) → src/core/executor/. - VERIFICATION.md: ALL remaining pre-core-split paths fixed (memory2/executor/orchestrator test commands + module pointers); ROADMAP tracer pointer; PHASE2-TRUST-LOOP's runbook command fixed (adapter.test.ts was deleted in W5 — removed) and PROVEN to resolve (34 tests). - Exhaustive residual sweep across *.md + docs/: zero non-historical pre-split paths remain. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Vsq543S6voPTUsGsEqZvnD * docs: exists-checked path sweep — src/main→core/orchestrator (codex S-A round 3) Round-3 findings + a proactive exists-check sweep: ROADMAP's configStore/memoryContext pointers, the eval fixtures/score header comments, and 12 stale src/main/<file> references across the md corpus (all moved to src/core/orchestrator in W7) — now verified by an automated every-md-path-must-exist check returning zero stale refs. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Vsq543S6voPTUsGsEqZvnD * docs: last stragglers — PHASE2 adapter-successor pointer + eval-source comment paths (codex S-A round 4) PHASE2-TRUST-LOOP prose now names facade.test.ts as the successor to the W5-deleted adapter.test.ts; every src/brain|src/executor comment path inside src/core/brain/eval/* swept to core (grep-verified zero non-core refs remain). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Vsq543S6voPTUsGsEqZvnD * docs: two path stragglers that escaped the exists-check shapes (codex S-A round 5) PHASE2's relative memory2/profileFacts pointer (no src/ prefix) and VERIFICATION's src/main/memoryHealth* wildcard — both shapes now covered in the sweep; ROADMAP's relative brain/index.ts ref fixed alongside. src/main/ipc*.test.ts verified VALID (ipc tests do live in src/main). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Vsq543S6voPTUsGsEqZvnD * docs: precise privacy claim — the index is derived/plaintext, the FILES are encrypted (codex S-A round 6) Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Vsq543S6voPTUsGsEqZvnD * docs: embedder-switch remedy is delete-derived-index-only, not auto-re-embed (codex S-A round 7) vectorCache.ts:20 IDENTITY REFUSAL: a mismatched (embedModel, dim) header THROWS; the store does not silently re-embed. Remedy: delete the derived index/ subdir (zero-authority; next launch re-embeds from the durable encrypted files). My round-2 rewrite overstated the automation. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Vsq543S6voPTUsGsEqZvnD * docs: the PHASE2 gate command now runs facade.test.ts, matching its own prose (codex S-A round 8 — cap) Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Vsq543S6voPTUsGsEqZvnD --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What & why
Makes roro's memory moat visible and honest on the default (codex) path, backed by a real proof that recalled memory actually shapes a real diff.
Grounded in a live experiment (real qwen2.5:3b + real codex): a marker-free "create hello.py" request, preceded by a recalled convention, put the marker into
args.taskand the written file 5/5, control 0/5. The moat mechanism works — so the receipt is now earned.The slice (5 commits)
run.completed.finalTextfrom the last agent_message (attachCodexFinalText/stitchCodexFinalText, mirrored in the runCodex loop + the proposal-eval fixture in lockstep). Previously only claude setfinalText(from its SDKm.result), so the truthful memory receipt was structurally dark on codex. Within-contract (finalText?is already optional onrun.completed), the pure mapper stays pure, boundary untouched.verify:memory-steered-real, opt-in/non-CI): asserts treatment reachesargs.taskand the created .py file honors the convention (marker as its header), ≥60% treatment / 0 control. Complements — does not replace — the deterministic fake-modelsmoke-memory-steered(packaged plumbing).runCodexloop coverage (mocked spawn over the captured fixture) — the production streaming path had none.Review
finalTextconsumers verified correct, the codex double-emit is exact parity with claude, no missed readers.Verification
tsc clean · 1099 passed / 12 skipped · eslint 0 errors. Live steering guard re-measured with the strict per-header check. Renderer changes are copy/comment-only (jsdom-covered); receipt render mechanics unchanged from S5.
🤖 Generated with Claude Code