chore(eval)/docs: signal-reconcile — instrument repair, deflect-on-vague fixtures, docs absorb the first dogfooding session#162
Merged
Conversation
…path family in CI Since the W7 core-split (#149) moved src/brain -> src/core/brain, both live runners still wrote/read their artifacts under the stale 'src/brain/eval' (runEval.ts latest/baseline writes; runProposalEval.ts baseline read + proposalLatest/proposalBaseline writes) and crashed ENOENT at the end of a full run. The directory now lives once in eval/paths.ts (EVAL_DIR + evalArtifactPath), both runners use it, and paths.test.ts pins that EVAL_DIR resolves to the directory the eval files actually live in (plus that the checked-in baseline.json exists there) — sabotage-verified: reverting EVAL_DIR to the stale path fails both tests. Also moves the two stale .gitignore scratch entries (latest.json / proposalLatest.json) to the real output location so runner scratch stays untracked. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…IDE fixtures (fixture-first)
First founder dogfooding session (2026-07-05): 'create something interesting'
made the 3B answer 'Let's brainstorm a fun project idea together' — a chat
deflection that fails ROADMAP §6's lands-or-clarifies bar (do the task or ask
a PERTINENT question). This adds the failure class to the eval WITHOUT
touching decide (measure first, fix later):
- DecideCase grows alsoAccept (a golden SET for lands-or-clarifies cases) +
a 'vague-initiative' tag (the DECIDE-side analogue of BehavioralTaxonomy).
- 5 fixtures (vague-interesting/-cool/-fun/-buildme/-impress) accept exactly
{run_agent, clarify}; answer FAILS. Expected to fail against the current
model — that is the point.
- scoreDecision gains the optional alsoAccept set (red-first unit tests: a
deflection still scores wrong_command).
- fixtures.test.ts pins the class deterministically in CI: acceptance set is
exactly {run_agent, clarify}, and every vague case must NOT be caught by
the deterministic clarify gate (same loud-re-label discipline as the
marker-less pin), so the fixtures keep measuring the MODEL.
Live tier stays opt-in/report-only; the deterministic suites stay green.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…eal Ollama, new fixtures included) OLLAMA_AVAILABLE=1 npm run eval:brain -- --write-baseline against local qwen2.5:3b — the sanctioned baseline workflow; baseline.json is never hand-edited. The old snapshot predated the W3 5→50 fixture expansion and the new runner output shape. Old → new headline numbers: - decide: 22/22 (100%) → 26/27 (96.3%; one wrong_command: run-writetest → answer — DECIDE streams at temperature 0.3, a run is a snapshot) - extract: 10/10 (100%) → 10/10 (100%), unchanged - behavioral value-quality: 2/5 (40%, the original 5-case set) → 26/35 (74.3%, the full expanded set — reproduces the #145 report-only number) - NEW keys now captured: behavioralNull 7/15 (46.7%; all 8 misses are hard-negative false_facts) + behavioralByTaxonomy (hard-negative 20% is the weak family; multi-fact/supersede/marker-less 100%) MEASURED TRUTH on the new vague-initiative cases: contrary to expectation, the current 3B lands-or-clarifies on all 5 (and 30/30 in a 3×-repeat probe with and without a memory section) — the fixtures therefore PIN the class against regression; the live 2026-07-05 deflection has not reproduced from paraphrased transcripts, so the fixture-block comment now records that and points the follow-up at promoting a RORO_TRACE capture of a real deflecting turn. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…, ROADMAP, VERIFICATION - LESSONS.md: new Dogfooding entry (2026-07-05) naming the four failure classes — deflect-on-vague / feedback-fade / working-illegibility / env-overrides-config — each with its file:line locus and fix status (2–4: the sibling dogfood-blocker slice; deflect-on-vague: fixture-first, with the measured 2026-07-06 twist that the paraphrased fixtures currently PASS, so the real repro needs a RORO_TRACE capture). - HANDOFF.md: §3 work log extended past W8 — the floating redesign (#153–#160, resting purity + truthful receipt), earn-the-moat (#161, codex finalText stitch + verify:memory-steered-real 5/5 vs 0/5), the first dogfooding session, and this signal-reconcile slice; §4 gains the refreshed-baseline numbers (the old bullets stay as the historical snapshot); §8 gains the corrected forward order (dogfood fixes → signing → legible-moment → executor-facts pilot → self-rehearsal → demo → stranger session). - docs/ROADMAP.md: Arc A §6 step 5 (founder dogfooding) marked STARTED 2026-07-05 with the four findings + the fixture-covered status of deflect-on-vague. - docs/VERIFICATION.md: the stale pre-W7 layout references in the cohort trace-review section (lines ~10–27) now point at src/core/… (the vitest command there was unrunnable). Stale paths in LATER sections of the file remain — out of this slice's scope, flagged in the handoff report. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…ing (S-A round 1) Panel findings (all CONFIRMED, 0 false positives across the 5-hat review): - BLOCKER: four docs-of-record sites claimed feedback-fade was 'being fixed in the sibling dogfood-blocker slice' — it is NOT (it belongs to the upcoming legible-moment slice). LESSONS, HANDOFF x2, ROADMAP now say so honestly, and working-illegibility is split truthfully (sleep-mid-run half → dogfood-blockers; the distinct working pose → legible-moment). - LESSONS locus corrected: summoning the ask also pokes the presence clock, not only petting. - alsoAccept untag loophole closed: any DECIDE case widening its acceptance set must declare a failure-class tag (pinned in fixtures.test.ts) — an untagged alsoAccept was silent softening. - evalArtifactPath now __dirname-based (correct from ANY cwd, travels with the dir on a move); new source-grep pin: the runners must route through evalArtifactPath, no quoted hand-spelled eval dirs (the exact post-W7 breakage class), header-comment convention exempted. - Stale src/brain path sweep: cohortTraceReview.ts fixture pointer, COHORT_TRACE_TO_EVAL.md verify command, RUN.md:20, README.md x2, WS1 doc x2, ROADMAP live locus. tsc clean · 1104 passed / 12 skipped · eslint 0 errors. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Vsq543S6voPTUsGsEqZvnD
…im (codex S-A round 1) Codex caught materially-false user-facing docs surviving the reconcile: README (6 sites) and RUN.md (4 sites) still described memory as an in-process PGlite+pgvector database at src/memory2 — deleted in W5 (#147); memory is encrypted files-as-truth + a rebuilt-on-launch in-memory vector index at src/core/memory2, embeddings via local nomic-embed-text. Also HANDOFF's eval header said 'temp 0' while decide actually streams at temperature 0.3 (brain/index.ts:190) — the same header the new baseline note's temp-0.3 explanation contradicted. And the crosslaunch proof pointer moved to src/core/orchestrator/. All claims now match the code. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Vsq543S6voPTUsGsEqZvnD
… advice (codex S-A round 2) - RUN.md's embedder note was DANGEROUS post-W5: it described a fixed vector(N) DB column and told users to move/delete the memory dir when changing embedders — that dir now holds the durable encrypted memories (files-as-truth); vectors.jsonl is the zero-authority derived cache that re-embeds automatically. Rewrote to the true architecture. - README executor pointers (table + tree) → src/core/executor/. - VERIFICATION.md: ALL remaining pre-core-split paths fixed (memory2/executor/orchestrator test commands + module pointers); ROADMAP tracer pointer; PHASE2-TRUST-LOOP's runbook command fixed (adapter.test.ts was deleted in W5 — removed) and PROVEN to resolve (34 tests). - Exhaustive residual sweep across *.md + docs/: zero non-historical pre-split paths remain. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Vsq543S6voPTUsGsEqZvnD
…-A round 3) Round-3 findings + a proactive exists-check sweep: ROADMAP's configStore/memoryContext pointers, the eval fixtures/score header comments, and 12 stale src/main/<file> references across the md corpus (all moved to src/core/orchestrator in W7) — now verified by an automated every-md-path-must-exist check returning zero stale refs. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Vsq543S6voPTUsGsEqZvnD
…e comment paths (codex S-A round 4) PHASE2-TRUST-LOOP prose now names facade.test.ts as the successor to the W5-deleted adapter.test.ts; every src/brain|src/executor comment path inside src/core/brain/eval/* swept to core (grep-verified zero non-core refs remain). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Vsq543S6voPTUsGsEqZvnD
… S-A round 5) PHASE2's relative memory2/profileFacts pointer (no src/ prefix) and VERIFICATION's src/main/memoryHealth* wildcard — both shapes now covered in the sweep; ROADMAP's relative brain/index.ts ref fixed alongside. src/main/ipc*.test.ts verified VALID (ipc tests do live in src/main). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Vsq543S6voPTUsGsEqZvnD
…ES are encrypted (codex S-A round 6) Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Vsq543S6voPTUsGsEqZvnD
…e-embed (codex S-A round 7) vectorCache.ts:20 IDENTITY REFUSAL: a mismatched (embedModel, dim) header THROWS; the store does not silently re-embed. Remedy: delete the derived index/ subdir (zero-authority; next launch re-embeds from the durable encrypted files). My round-2 rewrite overstated the automation. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Vsq543S6voPTUsGsEqZvnD
…wn prose (codex S-A round 8 — cap) Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Vsq543S6voPTUsGsEqZvnD
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What & why
The first founder dogfooding session (2026-07-05) produced roro's first real signal; this slice makes it measurable and makes the docs of record absorb it — no product behavior changes.
eval/paths.ts(__dirname-based) + CI pins: location, baseline-exists, and a source-grep pin that the runners route throughevalArtifactPath.vague-initiativeDECIDE fixtures (pass = {run_agent, clarify}; a deflection to answer fails) with analsoAcceptscorer mechanism, pinned so the acceptance set can't soften and widening requires a declared failure-class tag. Honest twist recorded: the fixtures passed 30/30 in probes — the live deflection doesn't reproduce from paraphrases, so they pin-against-regression; the real repro needs aRORO_TRACEcapture.--write-baseline, real Ollama): DECIDE 26/27 · behavioral 74.3% (full 50-case set) · NEW behavioralNull 46.7% + byTaxonomy keys. The instrument works again.Review
5-hat adversarial panel (15 confirmed findings, 0 false positives) → all fixed, incl. the blocker (docs claiming feedback-fade was in the sibling slice). Codex max-effort: 8 rounds, every finding real (materially-false memory claims → path families → precision nits), final round's fix applied + proven (
facade.test.tsruns in the gate command).Gate
tsc clean · 1104 passed / 12 skipped · eslint 0 errors · repaired runbook commands proven to resolve.
🤖 Generated with Claude Code
https://claude.ai/code/session_01Vsq543S6voPTUsGsEqZvnD