docs(branch): RFC for composed environment snapshotting + rollout branching - #1045
docs(branch): RFC for composed environment snapshotting + rollout branching#1045JeremyJC67 wants to merge 2 commits into
Conversation
![devin-ai-integration[bot]](https://avatars.githubusercontent.com/in/811515?s=60&v=4){kind=small}
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 3890653680
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| `env-ready` deliberately precedes `install_agent()` so skills-on/off branches re-run | ||
| skill deployment from a skill-free world (skills are baked in at install time). |
There was a problem hiding this comment.
When the source rollout uses bundled skills with a Dockerfile-built environment, this boundary is not skill-free: Rollout.setup() injects the skills into the Dockerfile (src/benchflow/rollout/__init__.py:1009-1016), and start() builds and launches that image before this snapshot. Consequently, an env-ready snapshot from a with-skill run already contains the skill files, so its no-skill child can access them even if install_agent() omits the discovery links, invalidating the proposed ablation. The design needs a skill-neutral build/runtime upload path or an explicit scrub before creating no-skill children.
AGENTS.md reference: AGENTS.md:L30-L30
Useful? React with 👍 / 👎.
3890653 to
80565f3
Compare
Specifies the three-layer checkpoint composition (environment x container x agent-session), stage-boundary snapshot policy mapped onto existing lifecycle phases, per-child branch deltas reusing the S/C/skill_mode axes, branch lineage artifacts (tree.json + per-child result.json + source_provenance kind=benchflow-branch), and the replay cut-point API. Tracks FrontierPhysics#73. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DZggU3vNYzr3hfDZJUE7jX
…ations §3.2 said the env-ready snapshot is a skill-free world, full stop. That holds only when the source rollout is no-skill: a with-skill run's setup() injects the bundled pack into the Dockerfile and start() builds that image before the env-ready snapshot is taken, so the snapshot already carries /skills. A no-skill child forked from it would restore the pack, deploy nothing on top of it, and still be labelled no-skill in every artifact. The engine already fails closed on exactly this (BranchParentSkillModeConflict unless the parent's own mode is no-skill); the RFC now states the precondition that gate enforces, so the design text and the implementation agree. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DZggU3vNYzr3hfDZJUE7jX
ecf4160 to
abee192
Compare
1 participant
Summary
Adds
docs/rollout-branching-rfc.md: a design for composed environment snapshotting + rollout branching, so a failed rollout can be attributed to a stage (env/tool init, research, execution, self-judgment) instead of reported as an undifferentiated zero, and so ablations (skills on/off, tool outage, plan injection) can be run as controlled comparisons rather than independent re-runs.Design-only PR; no behavior change. It composes what benchflow already ships rather than inventing a parallel mechanism:
Rollout.branch(n)already implements quiesce → checkpoint → fork → restore → aggregate, but checkpoints the Environment plane only;Sandbox.snapshot()/restore()contract from Sandbox snapshot/restore is exported as helper functions, not implemented on the Sandbox contract used by the kernel #384/fix: Sandbox.snapshot/restore on the contract used by kernel (#384) #470 exists, but the branch engine gates onsupports_snapshotand never calls it;docs/architecture.mdstates the gap directly: "container and agent-session checkpoint composition remain future work";bench eval continue's record-replay proxy is a working agent-session reconstruction bridge.The RFC specifies: the three-layer composed checkpoint and its ordering, stage boundaries mapped onto the existing lifecycle phases, per-child branch deltas reusing the existing S/C/
skill_modeaxes, branch lineage artifacts (branched runs currently leave none), and a replay cut-point API. Agent-session snapshotting is explicitly out of scope for v1, with the record-replay bridge named as the v1 substitute. Side effect: it makes the declared-but-fail-closedbranch_execution: forked-snapshotvalue intask-standard.mdreal.Test plan
uv run python -m pytest tests/ -q→5814 passed, 65 skipped(single pre-existing unrelated failure,test_check_results_accepts_symlinked_current_repo_inferred_source, fails identically on an untouched tree)uv run ruff check .→ All checks passed;uv run ty check src/→ All checks passedImplementation lives in a follow-up PR (draft) so the design can be discussed first. Tracks FrontierPhysics#73.