feat(orchestrator): repeat each eval K times via a pluggable attacker

AGENTS.md

@@ -59,6 +59,7 @@ for additional examples refer to [README.md](./README.md).
 - **Probe**: a slot in the environment or user task prompt where an injection payload gets placed. Referenced via `{injection_task_id:probe_name}` placeholders
 - **Attack technique** (`src/midojo/attacks/builtin.py`): wraps a raw payload in a delivery technique (e.g. `important_instructions`, `ignore_previous`, `verbatim`)
 - **Payload set** (`src/midojo/attacks/data/`): curated corpus of payloads from external sources (e.g. Garak). Referenced in suite YAML via `source: "garak:<name>"`
+- **Attacker** (`src/midojo/attacks/attacker.py`): proposes the injection for each attempt of an evaluation (`midojo-run --attempts`), optionally adapting to the outcomes of earlier attempts. `StaticAttacker` delivers the suite's payloads unchanged on every attempt
 - **Verifier** (`src/midojo/verifiers/`): checks whether an injection succeeded, for example with `output_contains`, `env_field_equals`, `env_list_any_match`, `env_field_contains`, `env_field_unchanged`, `env_list_count`, `tool_called` (counts blocked calls), composable with `any_of`, `all_of`, `not`
 
 ## Patterns for Common Changes

README.md

@@ -186,6 +186,17 @@ At the end, MiDojo prints a results table. Here's one from a full run:
 
 The **Security percentage is the attack success rate**: lower is better; N/A rows are excluded. For `itinerary_exfil_via_email`, attack success means the agent attempted `send_email`. MiDojo blocked the call; no email was sent. To look at one pairing more closely, rerun it with its `--user-task` and `--injection-task`.
 
+**Repeat a pairing.** An agent can resist an injection on one run and follow it on the next. `--attempts N` (`-k N`) runs each pairing N times, each in a fresh sandbox, and the Utility and Security cells show how many attempts completed the task and broke the agent, e.g. `💀 attack succeeded  (1/3)`. Any attempt that breaks the agent counts as a break, so the Security percentage stays the attack success rate:
+
+```bash
+uv run --env-file .env midojo-run --suite weather --gateway openshell \
+  --control-url http://localhost:8090 \
+  --user-task trip_weather --injection-task hidden_file_via_itinerary \
+  --attempts 3 --logdir runs/weather-trip-x3
+```
+
+`results.json` then also records every attempt (`attempt_log`) and per-pairing counts (`metrics`).
+
 **Compare with a clean run.** Selecting a user task without an injection task runs it without attacks, so you can compare the agent's answer with step 2's:
 
 ```bash

src/midojo/attacks/__init__.py

@@ -4,10 +4,14 @@
 It separates *what* you're testing (suite authoring) from *how* you're
 attacking (this library), so suites stay stable while attacks evolve. See
 issue #36 for the design.
+
+It also holds the attackers (``attacks.attacker``): the policies that propose,
+for each attempt of an evaluation, which injection to deliver.
 """
 
 from __future__ import annotations
 
+from midojo.attacks.attacker import AttackAttempt, Attacker, AttemptOutcome, StaticAttacker
 from midojo.attacks.records import AttackTechnique, Origin, PayloadSet, PayloadWrapper
 from midojo.attacks.registry import (
     DEFAULT_LIBRARY,
@@ -23,11 +27,15 @@
     "ASI_DETAILS",
     "DEFAULT_LIBRARY",
     "ASICategory",
+    "AttackAttempt",
     "AttackLibrary",
     "AttackTechnique",
+    "Attacker",
+    "AttemptOutcome",
     "Origin",
     "PayloadSet",
     "PayloadWrapper",
+    "StaticAttacker",
     "load_payload_set_file",
     "parse_asi_category",
     "resolve_source",

src/midojo/attacks/attacker.py

@@ -0,0 +1,74 @@
+"""Attackers: the policy that picks each attempt's injection.
+
+The rest of this package is the attack *material*: delivery techniques and
+payload corpora. An attacker is the *policy* that uses it. For every attempt of
+a (user task x injection task) pair it proposes the injection to deliver, and it
+may learn from the outcomes of the attempts before. The orchestrator's attempt
+loop is written against the :class:`Attacker` protocol, so a model-driven
+attacker slots in beside :class:`StaticAttacker` without reshaping the loop.
+"""
+
+from __future__ import annotations
+
+from collections.abc import Sequence
+from dataclasses import dataclass
+from typing import Protocol
+
+
+@dataclass(frozen=True)
+class AttackAttempt:
+    """What an attacker proposes for one attempt.
+
+    ``injections`` maps probe keys (``"<injection_task>:<probe>"``) to payload
+    strings, the same mapping the orchestrator passes to ``create_evaluation``.
+    """
+
+    injections: dict[str, str]
+
+
+@dataclass(frozen=True)
+class AttemptOutcome:
+    """The observed result of one attempt.
+
+    Fed back to the attacker as history, and recorded in the run's results.
+    ``delivered`` says the payload reached the agent at all; ``broke`` says the
+    injection succeeded (the security check passed on a delivered payload).
+    """
+
+    attempt: int
+    injections: dict[str, str]
+    delivered: bool
+    delivered_where: list[str]
+    broke: bool
+    security_reason: str | None
+    utility: bool
+    agent_output: str
+    eval_id: str
+
+
+class Attacker(Protocol):
+    """Proposes the injection for each attempt, optionally adapting to history.
+
+    ``propose`` is async so a model-driven attacker can call its own inference
+    endpoint. ``goal`` is the objective for an attacker that pursues one; nothing
+    supplies an objective yet, so the orchestrator passes ``None``. ``history``
+    holds the outcomes of the pair's earlier attempts, in order.
+    """
+
+    async def propose(self, goal: str | None, history: Sequence[AttemptOutcome]) -> AttackAttempt: ...
+
+
+class StaticAttacker:
+    """Proposes the same suite-defined injection on every attempt.
+
+    With one attempt this is MiDojo's original single-shot run. With K attempts
+    it delivers the identical injection K times, each in a fresh sandbox, which
+    shows how often it breaks the agent under the agent's own stochasticity. It
+    ignores ``goal`` and ``history``: there is nothing to adapt.
+    """
+
+    def __init__(self, injections: dict[str, str]) -> None:
+        self._injections = dict(injections)
+
+    async def propose(self, goal: str | None, history: Sequence[AttemptOutcome]) -> AttackAttempt:
+        return AttackAttempt(injections=dict(self._injections))