pi-evolver

A self-learning / auto-evolving mechanism for the Pi coding agent, implemented as a Pi extension (v0.84.x, tested on 0.84.2).

It closes the learn loop without hand-written rules:

tool calls ──► TRACE (traces.jsonl)
                 │
                 ▼
            MINE (recurring tool+verb failure patterns) ──► evolve.json
                 │
                 ▼
            QUARANTINE (≥ minFailures + high failure ratio ⇒ candidate)
                 │
        ┌────────┴─────────┐
        ▼                  ▼
   /evolver accept     /evolver reject      ← HUMAN GATE
   (always injected)   (silenced forever)
        │
        ├──► INJECT into system prompt (before_agent_start)
        │         └──────────────────────► shapes the next agent run
        │
        └──► PROMOTE to native Pi Skill (v0.2)
                  └──────────────────────► persists across sessions

How it works

• Trace — every tool_call/tool_result pair is logged to <store>/traces.jsonl as one JSON line: {ts, tool, isError, file, cmd, verb}. Failures are detected from the Pi API's isError flag first; a heuristic text-sniff (Error:, non-zero exit, command not found, …) is used only when the tool does not provide a definitive signal.
• Verb extraction — commands are normalized before keying: environment variable assignments (KEY=VALUE), known prefix commands (sudo, time, env, …), and subshell parentheses are stripped. Two-level verbs capture subcommands for CLIs like npm/git/docker (e.g. npm:test, git:commit). Chained commands (&&, ||, ;) use the last segment.
• Mine — after each agent_settled, events fold into <store>/evolve.json: a counter per pattern key tool:verb.
• Quarantine — once a pattern accumulates minFailures (default 2) and its failure ratio exceeds minFailureRatio (default 0.3), it becomes a candidate. Candidates are shown to the model tagged (candidate: unverified) so the model can act on them, but they are always visible to you and reversible. Candidates are ranked by failure ratio × recency (exponential decay, 7-day half-life).
• Recovery detection (v0.2) — when a failure on target X is followed by a success on target X using a different tool/verb within 3 turns, the recovery strategy is recorded (lastRecovery). This gives the system the semantic pair "failed behavior → successful substitute".
• Inject — on every before_agent_start, accepted lessons + the top candidates (within maxInjected/maxInjectedChars caps) are appended to the system prompt under a clearly labeled ## Pi self-learning (pi-evolver) block. Lessons include the most recent error snippet when available, giving the LLM concrete failure context.
• Skill promotion (v0.2) — accepted learnings with enough evidence (≥minFailuresForSkill failures, ≥minEventsForSkill total events) are compiled into native Pi SKILL.md files using a deterministic template (no LLM call). Skills include the pattern, error snippet, human note, recovery strategy, and contextual guidance. Pi discovers them via resources_discover on the next session start or /reload.
• Gate — promotion to always injected requires /evolver accept <id> [note]. /evolver reject <id> silences a pattern permanently (human veto). Nothing self-modifies its own code, and all learnings are deletable (it's JSON).

Install / uninstall

pi install git:github.com/yoda-digital/pi-evolver    # install
pi remove git:github.com/yoda-digital/pi-evolver     # uninstall

Or manually: clone to ~/.pi/agent/extensions/pi-evolver/ and npm install.

Storage & config

Item	Location
Learning store	~/.pi/evolver/evolve.json (override: PI_EVOLVER_DIR)
Raw traces	~/.pi/evolver/traces.jsonl (pruned after 30 days at session start)
Generated skills	~/.pi/evolver/skills/ (one SKILL.md per promoted learning)
Debug log	PI_EVOLVER_DEBUG=<file> — writes trace/mine/inject/promote steps

Tunables (defaults in src/config.ts):

Tunable	Default	Purpose
minFailures	2	Min failures before quarantine
minFailureRatio	0.3	Min fail ratio for quarantine
maxInjected	12	Max learnings in system prompt
maxInjectedChars	2400	Hard cap on injected text
traceRetentionDays	30	Trace pruning window
learningRetentionDays	90	Stale learning pruning window
minFailuresForSkill	3	Min failures before skill promotion
minEventsForSkill	5	Min total events before skill promotion
skillsSubdir	skills	Subdirectory for generated SKILL.md files
maxSkillBodyChars	2000	Max body length of compiled SKILL.md

Commands

Command	Description
/evolver status	Learning counts + top candidates
/evolver show <id>	Full record for a pattern (e.g. bash:npm:test)
/evolver accept <id> [note]	Promote to always-injected; note replaces the generic lesson
/evolver reject <id>	Drop a pattern from injection permanently
/evolver promote <id>	Compile to SKILL.md + reload (immediate activation)
/evolver skills	List generated skills on disk
/evolver demote <name>	Remove a generated skill
/evolver-lessons	Print exactly what is currently injected into the prompt

Testing & benchmarks

npm test                      # 102 tests (unit + effectiveness benchmark)
npm run report                # learning report from your live data
bash benchmark/live.sh        # pipeline verification (12 checks through real Pi)
bash benchmark/measure.sh     # effectiveness experiment (before/after comparison)
bash benchmark/measure.sh 8   # 8 runs instead of default 5
EVOLVER_CONTROL=1 bash benchmark/measure.sh  # control mode (no learning between runs)

Effectiveness benchmark (npm test)

5 realistic scenarios run through the evolver engine — cat-vs-read, npm-full-vs-focused, permission-denied, command-timeout, chained-command — measuring 8 metrics across pattern detection, recovery linkage, lesson quality, skill compilation, and cross-session injection:

pi-evolver EFFECTIVENESS BENCHMARK
════════════════════════════════════════════════════════════
  Scenarios tested:        5
  Pattern detection:       100%
  Verb accuracy:           100%
  Recovery linkage:        100%
  Snippet capture:         100%
  Lesson quality:          60%
  Skill generation:        20%
  Skill quality:           100%
  Cross-session warning:   100%
════════════════════════════════════════════════════════════

Live benchmark (benchmark/live.sh)

Runs real Pi sessions headlessly to verify the full pipeline end-to-end:

1. Triggers bash:cat failures via pi -p (3 headless prompts)
2. Verifies: events traced, pattern mined, snippet captured, candidate quarantined
3. Accepts + promotes the learning (triggers auto-skill-write)
4. Runs a fresh session, verifies: injection present, skill mentions the learned pattern

Checks 12 pipeline stages — all deterministic (verifies evolver data, not model behavior).

Effectiveness experiment (benchmark/measure.sh)

The grounded measurement: runs Pi on the same broken 3-bug project multiple times and measures per-run metrics. The evolver learns across runs.

The project has 3 deliberate bugs: wrong export name, missing .env file, wrong import path. Pi must find and fix all 3 for npm test and npm start to pass.

  Run     Tool Calls   Failures     Fail Rate   Time      Injected
  ───     ──────────   ────────     ─────────   ────      ────────
  #1      24           8            33.3%       45.2s     0ch
  #2      21           6            28.6%       38.1s     340ch
  #3      18           4            22.2%       31.5s     520ch
  #4      15           3            20.0%       28.3s     520ch
  #5      14           2            14.3%       25.1s     680ch

  Trend (first 2 runs vs last 3 runs):
    Early failures:  14  |  Late failures:  9
    Failure reduction: 36% — evolver learning is effective

Run it yourself: bash benchmark/measure.sh 5. Compare with the control (no learning): EVOLVER_CONTROL=1 bash benchmark/measure.sh 5.

Learning report (npm run report)

Analyzes your actual ~/.pi/evolver/evolve.json and prints a metrics dashboard: tool distribution, failure patterns, candidates, generated skills, and mistake elimination rate.

Safety posture

1. Learning data is plain JSON you can read, edit, and delete.
2. Injection is capped (count + chars) and framed as observed failure patterns.
3. Tracing is bounded output (≤300 chars) and failures never propagate into the agent loop.
4. The extension does not write code, does not grant tools, and does not call providers.
5. Accepted lessons are your own vetted statements — the human gate is the whole point.
6. Skill compilation is deterministic template expansion — no LLM call, no generated code.
7. Command verbs (first word(s) of commands) are included in injected lessons and sent to the model provider as part of the system prompt — be aware if working with sensitive tool names.
8. The DB is written atomically (write-to-tmp + rename); stale non-accepted learnings are auto-pruned after 90 days.
9. Generated skills are standard SKILL.md files — inspectable, editable, deletable.

Known limitations

• No file locking: concurrent Pi sessions sharing ~/.pi/evolver/ may overwrite each other's learning counts. Accepted learnings and generated skills are safe.
• Synchronous FS: the extension uses readFileSync/writeFileSync for simplicity. The files are small (a few KB) so this does not meaningfully block the event loop.
• Relative path recovery: cat ./foo.ts and read /home/user/foo.ts will not be linked as a recovery because relative paths are not resolved (too expensive).
• Skills appear next session: auto-promoted skills are written to disk but not reloaded mid-session. Use /evolver promote <id> for immediate activation.

Repo layout

pi-evolver/
├── index.ts            # Pi extension entry (events, commands, injection, skills)
├── src/config.ts       # tunables
├── src/trace.ts        # verb extraction, error sniffing, target normalization
├── src/evolve.ts       # DB, mining engine, selection, skill compiler (FS-injectable)
├── src/store.ts        # store dir resolution + trace/learning pruning
├── test/evolver.test.ts
├── SPEC-v0.2.md        # v0.2 engineering specification
└── vitest.config.ts