Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 7 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,13 @@

## [Unreleased]

### Added
- **Fystash sandbox backend.** `--sandbox fystash` implements the Sandbox
protocol over the public Fystash API (`compile` → Branch → Plan → apply).
Install the `sandbox-fystash` extra. Snapshot/restore is same-Run
`/workspace` tar rollback, not a Firecracker VM snapshot. Arbitrary guest
Dockerfiles, DinD, and GPU are residuals.

## 0.6.9 — 2026-08-15

### Changed
Expand Down
2 changes: 1 addition & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -93,7 +93,7 @@ uv tool install --python 3.12 --upgrade benchflow
in the install command so `uv` does not resolve an older Python-compatible
package that lacks the CLI entrypoints.
- If you see `Executables already exist: bench, benchflow`, re-run with `uv tool install --python 3.12 --upgrade --force benchflow` to replace stale entrypoints from an older install.
- For Daytona, Modal, or AgentCore extras, install the relevant optional package, for example `uv tool install --python 3.12 --upgrade 'benchflow[sandbox-daytona]'`.
- For Daytona, Modal, AgentCore, or Fystash extras, install the relevant optional package, for example `uv tool install --python 3.12 --upgrade 'benchflow[sandbox-daytona]'` or `benchflow[sandbox-fystash]`.

Internal users wanting the newest preview from `main` install the [internal preview channel](./docs/release.md) (`uv tool install --python 3.12 --prerelease allow --upgrade benchflow`).

Expand Down
2 changes: 1 addition & 1 deletion docs/reference/cli.md
Original file line number Diff line number Diff line change
Expand Up @@ -240,7 +240,7 @@ bench eval run --tasks-dir ./tasks --matrix matrix.yaml --trials 3
| `--agent` | `claude-agent-acp` | Agent name |
| `--model` | Agent default | Model ID |
| `--reasoning-effort` | — | Agent reasoning/thinking effort when the agent exposes one (e.g. `max`) |
| `--sandbox` | `docker` | Sandbox: docker, daytona, modal, apple-container, or agentcore |
| `--sandbox` | `docker` | Sandbox: docker, daytona, modal, apple-container, agentcore, or fystash |
| `--usage-tracking` | `auto` | Token usage telemetry policy: `auto`, `required`, or `off` |
| `--environment-manifest` | — | Environment-plane manifest applied to every rollout in the batch: a path to an `environment.toml`, or a `name@version` registry spec resolved via `$BENCHFLOW_ENV_REGISTRY` when set, else the built-in registry shipped with benchflow (`env0@prod`, `env0@outage`; see [Environment plane: Registry](../environment-plane.md#registry-nameversion)). Overrides a task.md `benchflow.environment.manifest` pin |
| `--state` | — | S-axis environment binding; inline JSON, registry `name@version`, or manifest path. Takes precedence over `--environment-manifest` |
Expand Down
5 changes: 5 additions & 0 deletions pyproject.toml
Original file line number Diff line number Diff line change
Expand Up @@ -88,6 +88,10 @@ sandbox-modal = [
"modal>=0.73",
"tenacity>=8.0",
]
sandbox-fystash = [
# Public Fystash Python SDK (catalog client + Revision/Branch/Plan/Run).
"fystash>=0.6.0",
]
sandbox-agentcore = [
# Bedrock AgentCore Runtime microVMs. The SDK supplies the interactive
# shell (`open_shell`) used as the ACP transport; boto3 supplies the
Expand Down Expand Up @@ -215,6 +219,7 @@ exclude = [
"src/benchflow/sandbox/daytona_strategies.py",
"src/benchflow/sandbox/daytona_dind.py",
"src/benchflow/sandbox/modal_impl.py",
"src/benchflow/sandbox/fystash.py",
"src/benchflow/sandbox/docker.py",
"src/benchflow/sandbox/_base.py",
"src/benchflow/_env_setup.py",
Expand Down
Loading