Isolate untrusted repository tests in fail-closed Docker containers

.github/workflows/docker-isolation.yml

@@ -0,0 +1,35 @@
+name: Docker isolation integration
+on:
+  pull_request:
+    paths:
+      - "backend/**"
+      - ".github/workflows/docker-isolation.yml"
+  push:
+    branches: [main]
+    paths:
+      - "backend/**"
+      - ".github/workflows/docker-isolation.yml"
+
+permissions:
+  contents: read
+
+jobs:
+  isolation:
+    runs-on: ubuntu-latest
+    timeout-minutes: 15
+    steps:
+      - uses: actions/checkout@v7
+      - uses: actions/setup-python@v7
+        with:
+          python-version: "3.12"
+          cache: pip
+          cache-dependency-path: backend/requirements.txt
+      - name: Install backend dependencies
+        run: python -m pip install -r backend/requirements.txt
+      - name: Fetch administrator-approved runner image
+        run: docker pull python:3.12-slim-bookworm
+      - name: Verify live Docker isolation constraints
+        env:
+          OPENAI_API_KEY: "CI_ONLY_DUMMY_KEY_NEVER_FORWARDED"
+          GITHUB_TOKEN: "CI_ONLY_DUMMY_KEY_NEVER_FORWARDED"
+        run: PYTHONPATH=backend python -m unittest backend/tests/docker_smoke.py -v

README.md

@@ -59,7 +59,7 @@ It is designed to sit **between an issue and a pull request** — with the devel
 
 - 🧠 Repository-aware task planning
 - ✍️ Structured code/file editing instead of trusting raw model-generated diffs
-- 🧪 Automatic test-command detection and execution
+- 🧪 Automatic test-command detection with fail-closed, isolated Docker execution
 - 🔁 Limited repair loop when tests fail
 - 🌿 **Automatic isolated Git branch per run**
 - ✅ **Explicit workspace commit API with per-commit author identity**
@@ -69,7 +69,7 @@ It is designed to sit **between an issue and a pull request** — with the devel
 - 🏠 Local-first LLM support with Ollama
 - 🔌 OpenAI/OpenRouter-compatible provider support
 - ⚡ FastAPI backend + lightweight web UI
-- 🐳 Docker support for the backend
+- 🐳 Local Docker sandbox for test execution (requires a separate local Docker daemon)
 
 ---
 
@@ -420,23 +420,51 @@ without stronger isolation and authentication.
 
 ## 🔐 Security model
 
-PatchPilot is an **MVP and local developer tool**. Do not expose the current backend directly to the public internet.
+PatchPilot **does not execute repository test scripts on the host**. Model-
+selected commands and `POST /api/execute` are dispatched to a separate,
+ephemeral Docker container using the [isolation policy](backend/app/sandbox.py).
+If Docker or the approved test image is unavailable, test execution **fails
+closed** instead of falling back to local execution.
 
-The agent can clone repositories and execute detected project commands inside a workspace. Before using PatchPilot in an untrusted or multi-user environment, add:
+The container runner uses:
 
-- container or VM isolation
-- CPU, memory, disk, and execution-time limits
-- network isolation
-- authenticated API access
-- command and path allowlists
-- secret isolation
-- explicit human approval before write/push/PR actions
+- No network access, no capabilities, no privilege escalation, a non-root user
+  and a read-only root filesystem
+- Read-only bind mount of the current repository; no host HOME, credentials,
+  Docker socket, or parent workspace directories mounted
+- Bounded memory (512 MiB), CPU (1 core), process count (64), temporary disk
+  (64 MiB) and execution time (180 seconds by default)
+- Bounded log capture, forced container cleanup after timeout and a strict
+  command parser that never invokes a shell
+- Preloaded test images (`docker pull python:3.12-slim-bookworm` for Python);
+  `--pull=never` keeps runtime execution offline
 
-GitHub issue import also introduces a credential boundary: keep `GITHUB_TOKEN` server-side and use the minimum permissions required.
+**Quick check (local):**
 
-The project roadmap intentionally includes a stronger sandbox for this reason.
+```bash
+docker pull python:3.12-slim-bookworm
+python -m pip install -r backend/requirements.txt
+PYTHONPATH=backend python -m unittest backend.tests.test_sandbox -v
+```
+
+For deterministic live Docker boundary checks, CI executes
+[backend/tests/docker_smoke.py](backend/tests/docker_smoke.py) with a real
+container. Python `unittest` works in the base image. Project test suites
+requiring third-party packages need an **administrator-prepared offline
+test image** (e.g. `PATCHPILOT_IMAGE_PYTHON=your-org/approved-image@sha256:...`).
+Repository files and the model cannot select arbitrary images.
+
+**Important limits:** Docker-based isolation is stronger than an executable
+allowlist but not a microVM, a proof of kernel confinement, or authentication.
+Cloning, Git actions and the agent backend still run on the local host. Keep
+the backend loopback-only, do not mount the Docker socket into the backend
+container, and do not expose this unauthenticated tool on the public internet.
+The Compose backend is loopback-published for safety but does not have access
+to a Docker daemon, so test execution from Compose fails closed by default.
+
+See [security boundaries and threat model](docs/security-boundaries.md) and
+[the tracked isolation requirements](https://github.com/harshitethic/PatchPilot/issues/23).
 
----
 
 ## 🗺️ Roadmap
 
@@ -462,7 +490,8 @@ The project roadmap intentionally includes a stronger sandbox for this reason.
 - [ ] GitHub App authentication
 - [ ] Streaming agent events
 - [ ] File-aware tool calling
-- [ ] Strong sandboxing with Docker / microVMs
+- [x] Fail-closed isolated Docker test runner with resource limits and CI smoke tests
+- [ ] Hardened multi-tenant microVM isolation, auth and deployment threat review
 - [ ] Human approval gates
 - [ ] Persistent runs and history
 - [ ] CLI: `patchpilot run "Fix the auth bug"`

backend/app/main.py

@@ -17,6 +17,7 @@
 from pydantic import BaseModel, Field
 
 from app.command_policy import parse_safe_command
+from app.sandbox import run_test_command
 
 APP_NAME = "PatchPilot"
 ROOT = Path(__file__).resolve().parents[2]
@@ -675,10 +676,10 @@
     for iteration in range(1, req.max_iterations + 1):
         if not test_argv:
             break
-        test_code, test_output = run(test_argv, repo, 180)
+        test_code, test_output = run_test_command(test_argv, repo, 180)
         last_test_code, last_test_output = test_code, test_output
         history.append({"iteration": iteration, "action": "test", "returncode": test_code, "output": test_output[-5000:]})
-        if test_code == 0 or iteration >= req.max_iterations:
+        if test_code == 0 or test_code in {124, 125, 126} or iteration >= req.max_iterations:
             break
 
         current_context = read_repo_context(repo, list_files(repo))
@@ -693,21 +694,22 @@
 
     diff_text = build_patch(repo)
     _, status = run(["git", "status", "--short", "--branch"], repo, 30)
     return {
         "workspace_id": workspace_id,
         "repo_name": safe_repo_name(req.repo_url),
         "base_branch": base_branch,
         "branch": requested_branch,
         "summary": plan.get("summary", ""),
         "plan": plan.get("plan", []),
         "touched_files": plan.get("touched_files", []),
         "test_command": test_command,
         "tests_passed": bool(test_argv) and last_test_code == 0,
         "test_output": last_test_output[-12000:],
+        "test_execution_backend": "docker" if test_argv else None,
         "diff": diff_text,
         "history": history,
         "git_status": status.strip(),
     }
 
 
 @app.post("/api/execute")
@@ -717,8 +719,8 @@
         argv = parse_safe_command(req.command)
     except ValueError as exc:
         raise HTTPException(400, f"Command blocked by PatchPilot safety policy: {exc}") from exc
-    code, output = run(argv, repo, 180)
-    return {"returncode": code, "output": output[-12000:]}
+    code, output = run_test_command(argv, repo, 180)
+    return {"returncode": code, "output": output[-12000:], "execution_backend": "docker"}
 
 
 @app.post("/api/workspace")

backend/app/sandbox.py

@@ -0,0 +1,163 @@
+"""Fail-closed execution of repository test commands.
+
+The backend itself is a trusted local process; repository test code is not.
+Tests run only inside disposable Docker containers with strict resource limits.
+This does not make Git cloning, Docker daemon access, or all kernel attack
+surfaces completely safe; consult docs/security-boundaries.md.
+"""
+from __future__ import annotations
+
+import os
+import re
+import secrets
+import selectors
+import shutil
+import subprocess
+import time
+from pathlib import Path
+
+# Trusted administrators may install private, prebuilt offline images by setting
+# the relevant env var. Repository files and model output cannot set these.
+DEFAULT_IMAGES = {
+    "python": "python:3.12-slim-bookworm",
+    "node": "node:22-bookworm-slim",
+    "go": "golang:1.23-bookworm",
+    "rust": "rust:1.87-bookworm",
+}
+IMAGE_ENV = {
+    "python": "PATCHPILOT_IMAGE_PYTHON",
+    "node": "PATCHPILOT_IMAGE_NODE",
+    "go": "PATCHPILOT_IMAGE_GO",
+    "rust": "PATCHPILOT_IMAGE_RUST",
+}
+IMAGE_PATTERN = re.compile(r"[A-Za-z0-9][A-Za-z0-9._:/@-]{0,250}\Z")
+EXECUTION_TO_PROFILE = {
+    "pytest": "python", "python": "python", "python3": "python",
+    "ruff": "python", "mypy": "python", "pyright": "node",
+    "npm": "node", "pnpm": "node", "yarn": "node",
+    "go": "go", "cargo": "rust",
+}
+MAX_OUTPUT_CHARS = 12000
+
+
+def select_image(argv: list[str]) -> str:
+    """Select an administrator-approved image, never one from repo contents."""
+    if not argv or argv[0] not in EXECUTION_TO_PROFILE:
+        raise ValueError("Unapproved test executable")
+    profile = EXECUTION_TO_PROFILE[argv[0]]
+    image = os.getenv(IMAGE_ENV[profile], DEFAULT_IMAGES[profile])
+    if not IMAGE_PATTERN.fullmatch(image) or ".." in image:
+        raise ValueError("Invalid configured test image")
+    return image
+
+
+def build_container_command(
+    argv: list[str], repo: Path, *, name: str | None = None
+) -> list[str]:
+    """Create a test-only Docker invocation without a shell or host credentials."""
+    if not argv or any(not isinstance(arg, str) or not arg or "\x00" in arg for arg in argv):
+        raise ValueError("Invalid test command argument")
+    image = select_image(argv)
+    resolved = repo.resolve(strict=True)
+    if not resolved.is_dir() or repo.is_symlink():
+        raise ValueError("Repository is not a real directory")
+    # Docker's --mount syntax is comma-delimited. Refuse paths it cannot quote
+    # securely instead of allowing mount-option interpretation.
+    if any(c in str(resolved) for c in (",", "\n", "\r", "=")):
+        raise ValueError("Unsupported workspace path for a safe bind mount")
+    container_name = name or "patchpilot-" + secrets.token_hex(12)
+    if not re.fullmatch(r"patchpilot-[a-f0-9]{24}", container_name):
+        raise ValueError("Invalid container name")
+    return [
+        "docker", "run", "--rm", "--init", "--pull=never",
+        "--name", container_name,
+        "--network", "none",
+        "--cap-drop", "ALL",
+        "--security-opt", "no-new-privileges",
+        "--pids-limit", "64",
+        "--memory", "512m", "--cpus", "1.0",
+        "--read-only", "--user", "65534:65534",
+        "--tmpfs", "/tmp:rw,nosuid,nodev,size=64m",
+        "--workdir", "/workspace",
+        "--mount", f"type=bind,source={resolved},target=/workspace,readonly",
+        "--env", "HOME=/tmp",
+        "--env", "CI=1",
+        "--env", "PYTHONDONTWRITEBYTECODE=1",
+        "--env", "PYTEST_ADDOPTS=-p no:cacheprovider",
+        "--env", "NPM_CONFIG_CACHE=/tmp/npm-cache",
+        image, *argv,
+    ]
+
+
+def docker_client_env() -> dict[str, str]:
+    """Never pass model/API tokens or redirect to arbitrary Docker daemons."""
+    allowed = ("PATH", "HOME", "USER", "TMPDIR", "LANG", "LC_ALL")
+    return {name: os.environ[name] for name in allowed if name in os.environ}
+
+
+def run_test_command(argv: list[str], repo: Path, timeout: int = 180) -> tuple[int, str]:
+    """Stream and cap untrusted output; never fall back to host execution.
+
+    Exit 125 = Docker unavailable, 124 = timeout, 126 = policy denial.
+    """
+    if not shutil.which("docker"):
+        return 125, "Test skipped: Docker isolation is required and unavailable."
+    name = "patchpilot-" + secrets.token_hex(12)
+    try:
+        cmd = build_container_command(argv, repo, name=name)
+    except (OSError, ValueError) as exc:
+        return 126, "Test blocked by container security policy: " + str(exc)
+
+    process: subprocess.Popen[bytes] | None = None
+    expired = False
+    chunks = b""
+    try:
+        process = subprocess.Popen(
+            cmd, stdin=subprocess.DEVNULL, stdout=subprocess.PIPE,
+            stderr=subprocess.STDOUT, shell=False, env=docker_client_env(),
+        )
+        assert process.stdout is not None
+        with selectors.DefaultSelector() as ready:
+            ready.register(process.stdout, selectors.EVENT_READ)
+            deadline = time.monotonic() + timeout
+            while True:
+                seconds = deadline - time.monotonic()
+                if seconds <= 0:
+                    expired = True
+                    break
+                events = ready.select(min(seconds, 0.5))
+                if not events:
+                    if process.poll() is not None:
+                        # Wait briefly for any final buffered stdout or EOF.
+                        continue
+                    continue
+                data = os.read(process.stdout.fileno(), 65536)
+                if not data:
+                    break
+                chunks = (chunks + data)[-MAX_OUTPUT_CHARS:]
+        if expired:
+            process.kill()
+        process.wait(timeout=5)
+        if expired:
+            return 124, "Container test timed out and was terminated."
+        return process.returncode or 0, chunks.decode("utf-8", errors="replace")
+    except (OSError, subprocess.SubprocessError):
+        return 125, "Docker test runner is unavailable; no host fallback was attempted."
+    finally:
+        if process is not None and process.poll() is None:
+            process.kill()
+            process.wait(timeout=5)
+        if process is not None and process.stdout is not None:
+            process.stdout.close()
+        # Always clean by a high-entropy exact container name, including
+        # timeout, client failures and successful completion.
+        if process is not None:
+            try:
+                subprocess.run(
+                    ["docker", "rm", "--force", name],
+                    stdin=subprocess.DEVNULL, stdout=subprocess.DEVNULL,
+                    stderr=subprocess.DEVNULL, timeout=15, check=False, shell=False,
+                    env=docker_client_env(),
+                )
+            except (OSError, subprocess.SubprocessError):
+                pass

backend/tests/docker_smoke.py

@@ -0,0 +1,36 @@
+"""Live Docker isolation smoke test. Run in CI on an Ubuntu Docker runner."""
+from __future__ import annotations
+
+import os
+import subprocess
+import tempfile
+import unittest
+from pathlib import Path
+
+from app.sandbox import run_test_command
+
+
+class DockerIsolationSmoke(unittest.TestCase):
+    def test_real_container_boundaries(self) -> None:
+        with tempfile.TemporaryDirectory() as root:
+            workspace = Path(root) / "workspace"
+            (workspace / "tests").mkdir(parents=True)
+            fixture = Path(__file__).parent / "sandbox_fixture" / "test_isolation.py"
+            (workspace / "tests" / "test_isolation.py").write_bytes(fixture.read_bytes())
+            # A real host secret sibling. A container with only the workspace
+            # mounted must not be able to read it.
+            (Path(root) / "patchpilot-host-secret.txt").write_text("DO_NOT_EXPOSE")
+            # Folder readable by nobody in the runner.
+            workspace.chmod(0o755)
+            (workspace / "tests").chmod(0o755)
+            code, output = run_test_command(
+                ["python", "-m", "unittest", "discover", "-s", "tests", "-v"],
+                workspace, timeout=45,
+            )
+            self.assertEqual(code, 0, output)
+            self.assertIn("Ran 4 tests", output)
+            self.assertNotIn("DO_NOT_EXPOSE", output)
+
+
+if __name__ == "__main__":
+    unittest.main()