Skip to content

feat: add independent bench and site cloning - #557

Open
BreadGenie wants to merge 51 commits into
developfrom
feat/agent-workspaces
Open

BreadGenie wants to merge 51 commits into
developfrom
feat/agent-workspaces

Conversation

@BreadGenie

@BreadGenie BreadGenie commented Oct 2, 2026 •

Copy link
Copy Markdown
Member

Summary

Add separate commands and Admin actions for copying bench code and site data into independent destinations.

pilot clone-bench uat
  copies app repositories, dependencies and assets; assigns fresh runtime ports
pilot clone-site source.localhost uat.localhost --target-bench uat
  creates a fresh database/login and independent public/private uploads

Bench cloning supports origin-default, named and current-checkout branches, including per-app overrides. Site cloning supports MariaDB and PostgreSQL, preserves the document encryption key, resets credentials, pauses the scheduler and disables outgoing mail. Database import and upload copying run concurrently, with rollback on failure. Compatible dependency and asset caches speed up bench cloning.

Related: #620 sets up a fresh development bench from an app checkout. The standalone clone-site command provides an explicit way to bring existing site data into a compatible destination bench. Fork, prepared-template and managed-worktree features are deferred.

Evidence

clone-bench and clone-site demo:

VID_20261002_162224.mp4
  • Before: the branch also exposed fork and prepare-template, combining checkout/environment creation with site-data copying.
  • After: only clone-bench and clone-site remain. CLI discovery, independent repository/file copying, cache reuse, database streaming, rollback and concurrent Redis ownership are covered by tests.
  • Validation: 3,561 tests passed, one skipped. Ruff, mypy, changed-function complexity checks and pre-commit passed.

Merge Danger

Door: two-way

New destinations are created explicitly; source code and site data are preserved. Failed site copies roll back their generated resources; failed bench copies remain available for inspection.

Blast Radius: moderate

Adds cloning commands and Admin actions, plus shared dependency/build caching. Destination apps and database engines must support the copied site. No worktree creation or whole-directory-copy POC is enabled.

Add pilot fork with source inference from the current bench, fresh site snapshots, independent Git worktrees, and isolated databases and files. Support reusable fixture templates and overlap artifact copying with database restore.

Serialize bench creation, reserve all runtime ports, and preserve worktree registrations during app moves and cleanup.
Copy app code with filesystem cloning where supported, materialize independent Git repositories, and recreate editable Python environments. Stream fresh MariaDB/PostgreSQL database copies directly into separate databases, copy uploads and default UAT sites to paused scheduling and muted email. The fresh fork command composes these primitives; prepared worktrees remain explicit.

Add authenticated Admin clone actions, destination selection and existing task progress. Validate isolation, stream failures, authorization, idempotency and conflicts. Targeted tests: 123 passed; frontend build and 73 tests passed. Full suite: 3008 passed, 1 skipped, with the existing template-fuzzer failure. Real MariaDB UI/CLI and source-write isolation verified.
Use source Git credentials for branch discovery and fetching, discard copied worktree registrations, and prepare nested frontend dependencies before builds. Respect site database connection settings and resolve Homebrew PostgreSQL clients. Return active idempotent clone tasks before validating destination existence.

Validation: 421 focused tests passed; changed-file lint, formatting and whitespace checks passed. Full suite: 3044 passed, 1 skipped, with one existing template fuzz failure reproduced on develop.
@BreadGenie
BreadGenie requested a review from tanmoysrt October 2, 2026 09:54
@greptile-apps

greptile-apps Bot commented Oct 2, 2026 •

Copy link
Copy Markdown
Contributor

RetriggerView in GreptileConfidence Score: 5/5

[High impact] The reviewed changes appear safe to merge.

Reviews (34) · Last reviewed commit: "fix: validate clone databases and preser..." · Reviewed by Greptile

Comment thread pilot/core/site/clone.py Resolved
Comment thread pilot/core/site/template.py Resolved
Comment thread pilot/core/site/clone.py Resolved
Comment thread pilot/core/site/clone.py Resolved
Comment thread pilot/core/site/clone.py Resolved
Comment thread pilot/managers/node_dependencies.py Resolved
Comment thread pilot/core/site/clone.py Resolved
Comment thread pilot/core/site/clone_database.py Resolved
Comment thread pilot/core/bench/cloning/bench.py Resolved
Comment thread pilot/core/site/clone_database.py Resolved
Comment thread pilot/core/bench/cloning/branches.py Resolved
Comment thread tests/pilot/core/test_cloning.py Resolved
Comment thread pilot/managers/node_cache.py Resolved
Comment thread pilot/managers/node_cache.py Resolved
@frappe-guard

frappe-guard Bot commented Oct 21, 2026 •

Copy link
Copy Markdown

Guard · Confidence 1/5 (apex) · risk high · 9 new findings (9 P1) · 13 open from earlier reviews (13 P1)

Affects 713 callers in 190 files. 9 callers may break.

Summary
  • Adds independent bench and site cloning through CLI, Admin tasks, and UI actions.
  • Supports branch selection, fresh site credentials, database streaming, rollback, and reusable dependency and asset caches.
  • Updates cloning documentation and adds workflow tests.
Confidence and Risk
  • The open site-scope finding permits unauthorized cloning of another site's data, and posted findings also identify unreliable database snapshots and asset builds.
  • Risk: high: Cloning spans shared runtime, database, filesystem, task, and Admin UI boundaries.
Findings
Severity File Finding
🟠 P1 pilot/managers/python_assets.py:81 Do not fail forced builds on cache fingerprint errors
🟠 P1 pilot/managers/python_assets.py:88 Do not fail successful builds during cache publication
🟠 P1 pilot/core/site/clone.py:79 Do not claim a route when registration returns none
🟠 P1 pilot/core/site/clone_database.py:86 Parallel schema dumps can lose a consistent snapshot
🟠 P1 pilot/core/bench/build_artifacts.py:220 Reject cache manifests missing required build outputs
🟠 P1 pilot/core/bench/cloning/bench.py:153 Do not copy source Git hooks into independent clones
🟠 P1 pilot/core/bench/artifacts.py:23 Avoid copying through a symlinked destination directory
🟠 P1 pilot/core/site/clone.py:176 Reject upload symlinks that escape the source site
🟠 P1 admin/frontend/dashboard/src/components/benches/BenchSwitcherDialog.vue:142 Only offer cloning for the current bench
🟠 P1 admin/backend/api/v1/cloning.py:104 clone_site route has no site scope check on the source (earlier review)
🟠 P1 pilot/core/bench/creator.py:100 Port scan drops the per-sibling guard, one bad bench.toml blocks creation (earlier review)
🟠 P1 admin/backend/api/v1/cloning.py:111 clone_site skips bench-management guard for the current bench (earlier review)
🟠 P1 pilot/core/app/node_packages.py:142 Yarn installs without a stamp are never considered installed (earlier review)
🟠 P1 pilot/core/bench/cloning/bench.py:108 Branch update raises StopIteration and skips the config write (earlier review)
🟠 P1 admin/backend/api/v1/cloning.py:54 clone_bench skips the bench-management guard its sibling applies (earlier review)
🟠 P1 pilot/core/bench/creator.py:100 Unreadable sibling bench config now aborts bench creation (earlier review)
🟠 P1 admin/backend/api/v1/cloning.py:104 Site clone route skips the site-scope check its siblings use (earlier review)
🟠 P1 pilot/core/app/__init__.py:287 Staging cleanup now raises on leftovers, aborting the clone (earlier review)
🟠 P1 admin/backend/api/v1/cloning.py:116 Random admin password breaks clone-site idempotency lookup (earlier review)
🟠 P1 admin/frontend/dashboard/src/components/common/CloneDialog.vue:9 CloneAppBranches/CloneBranchOptions types do not exist in the frontend (earlier review)
🟠 P1 admin/backend/api/v1/sites/core.py:157 Clone route skips require_scope, so site-scoped sessions can clone any site (earlier review)
🟠 P1 pilot/core/bench/cloning/bench.py:122 Clone copies source MariaDB/Postgres ports into destination config (earlier review)
Impact
flowchart LR
  n0["route:POST benches_bp:*/actions/clone"]:::changed
  n1["clone_bench"]:::changed
  n2["route:GET benches_bp:*/clone-branch-options"]:::changed
  n3["clone_branch_options"]:::changed
  n4["get_clone_bench_name"]:::changed
  n5["get_clone_site_names"]:::changed
  n6["clone_site"]:::changed
  n7["route:POST sites_bp:*/actions/clone"]:::changed
  n8["CloneAppBranches"]:::changed
  classDef changed fill:#fff3cd,stroke:#d39e00
  classDef external fill:#e7f1ff,stroke:#0b5ed7
  classDef tests fill:#f1f3f5,stroke:#adb5bd,color:#495057
Loading
Diagram
sequenceDiagram
    participant UI as Admin UI
    participant API as Admin API
    participant Task as Clone task
    participant Core as Bench/Site clone
    UI->>API: Submit clone request
    API->>Task: Queue clone
    Task->>Core: Clone destination
    Core-->>Task: Result or rollback
    Task-->>UI: Task status
Loading

Reviews (9) · Last reviewed commit: feat: fork development benches into isolated agent workspaces · Re-trigger · Fix all

Comment thread admin/backend/api/v1/sites/core.py Resolved
Use each source site's database engine for validation and destination credentials, regardless of either bench's default. Canonicalize source-site task reservations, follow boolean naming conventions and extract Redis test setup/cleanup to meet the complexity limit.

Keep clone routes bench-scoped: creating a destination site exceeds a source-site token's authority. Document and test this intentional access contract.
Comment thread pilot/core/bench/creator.py Resolved
Comment thread admin/backend/api/v1/cloning.py Resolved
Comment thread pilot/core/app/__init__.py Resolved
Comment thread pilot/core/app/__init__.py Resolved
Comment thread pilot/core/site/clone.py Resolved
Comment thread pilot/core/site/clone.py Resolved
Comment thread pilot/core/site/__init__.py Resolved
Comment thread pilot/core/app/__init__.py Resolved
Comment thread admin/backend/api/v1/cloning.py Resolved
Comment thread pilot/core/site/clone.py Resolved
Comment thread pilot/core/bench/cloning/bench.py Resolved
Comment thread pilot/tasks/clone_bench.py Resolved
Comment thread admin/backend/api/v1/sites/core.py Resolved
Comment thread pilot/internal/cli/fields.py Resolved
@BreadGenie

Copy link
Copy Markdown
Member Author

@frappe-guard rereview

Comment thread pilot/core/bench/cloning/bench.py Resolved
Comment thread pilot/core/bench/cloning/bench.py Resolved
Comment thread pilot/core/site/__init__.py Resolved
Comment thread pilot/internal/cli/fields.py Resolved
Comment thread admin/backend/api/v1/cloning.py Resolved
Comment thread pilot/core/site/clone_database.py Resolved
Comment thread pilot/core/app/__init__.py Resolved
Comment thread pilot/managers/node_dependencies.py Resolved
Comment thread admin/backend/api/v1/cloning.py Resolved
Comment thread admin/backend/api/v1/cloning.py Resolved
Comment thread pilot/core/bench/creator.py Resolved
Comment thread pilot/core/app/node_packages.py Resolved
Comment thread pilot/core/site/clone.py Resolved
Comment thread pilot/core/bench/build_artifacts.py Resolved
Comment thread pilot/core/bench/build_artifacts.py Resolved
Comment thread pilot/core/site/clone.py Resolved
@tanmoysrt

Copy link
Copy Markdown
Member

@guard review

Comment thread pilot/managers/python_assets.py Resolved
Comment thread pilot/managers/python_assets.py Resolved
Comment thread pilot/core/site/clone.py Resolved
Comment thread pilot/core/site/clone_database.py Resolved
Comment thread pilot/core/bench/build_artifacts.py Resolved
Comment thread pilot/core/bench/cloning/bench.py Resolved
Comment thread pilot/core/bench/artifacts.py Resolved
Comment thread pilot/core/site/clone.py Resolved
@BreadGenie

Copy link
Copy Markdown
Member Author

@frappe-guard review

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants