diff --git a/README.md b/README.md index ecc6959..746ec92 100644 --- a/README.md +++ b/README.md @@ -115,7 +115,7 @@ claude mcp add openadapt -- uvx --python 3.12 --from 'openadapt-agent[tutorial] When the server starts, Flow records the workflow once and, if needed, downloads a browser (about 300 MB). The first request waits for that. In our test on a Mac, the first request took about 80 seconds. The call returned `running` after 60 seconds, and the agent called `get_run` for the result. If setup takes more than 15 minutes, the request ends `did_not_run`. With Flow, a `done` result has `proof: "local"` instead of `"simulated"`. -Until 2.0.2 is on PyPI, use the command above even if your agent suggests a shorter one. The shorter command installs 2.0.1, which has no sandbox. +Until 2.0.2 is on PyPI, use the command above. The simulated sandbox suggests the same command in its `list_workflows` result. A command that installs `openadapt-agent[tutorial]` from PyPI gets 2.0.1, which has no sandbox: the server exits with an error that `--bundles` is required. ## What your agent gets back @@ -150,7 +150,7 @@ The check that stops a second entry covers one runs folder. See `--runs-dir` und - `what_happened` and `next_action`: fixed sentences you can show a person. They never contain input values or screen text. - `proof`: `local` for a `done` that was checked on your customer's computer, `simulated` for a sandbox run that didn't open the app, and `none` for every other outcome. - `model_calls`: on runs that reached Flow, how many AI model calls the run made. It's 0 unless model grounding, a fallback that asks an AI model where to click, is turned on (see [Security](#security)). -- `record_changed`: `yes`, `no`, or `unknown`. With `request_id_conflict`, its `no` describes this call only. The earlier request may have saved. +- `record_changed`: `yes`, `no`, or `unknown`. It's `no` only when OpenAdapt can show that nothing was written. `request_id_conflict` gives `unknown`, because the earlier request may have saved. ## Three tools @@ -311,9 +311,12 @@ Card text reaches your agent's model, so keep patient details out of it. The ser `openadapt-agent emit-skill BUNDLE --out ~/.claude/skills` writes a `SKILL.md` for one workflow: what it does, its inputs, how to call `run_workflow`, and what to do after each outcome. The skill holds no recorded values or secrets, and you still add the server separately. `--include-bundle` also copies the bundle next to the skill. That copy can contain values from the recording, so keep it as private as the recording. -Each skill is named after its workflow, so `enter_referral` becomes `enter-referral`. If the name would come out as `computer-use`, emit-skill uses `openadapt-gui-write` instead, so an assistant doesn't load the skill for every request to control the screen. +Each skill is named after its workflow, so `enter_referral` becomes `enter-referral`. Two names are reserved: -This repository also ships a general skill for any workflow, [skills/openadapt-gui-write/SKILL.md](skills/openadapt-gui-write/SKILL.md). A skill emitted under the `openadapt-gui-write` name into the same folder replaces it. +- `computer-use`, because an assistant would load that skill for every request to control the screen. +- `openadapt-gui-write`, the general skill for any workflow that this repository ships at [skills/openadapt-gui-write/SKILL.md](skills/openadapt-gui-write/SKILL.md). + +If a workflow's name would come out as either one, emit-skill names the skill `openadapt-` plus the workflow's id, such as `openadapt-workflow-1a2b3c4d5e6f7a8b9c0d1e2f`. An emitted skill never replaces the general skill in the same folder. ### Record a first workflow diff --git a/docs/DESIGN.md b/docs/DESIGN.md index 91a805b..24c17cd 100644 --- a/docs/DESIGN.md +++ b/docs/DESIGN.md @@ -117,7 +117,7 @@ makes: | `done` | Saved and checked by reading the record back | false | | `needs_review` | Stopped before saving; nothing written; a person decides | false | | `not_sure_if_saved` | A save may have gone through; a person checks the record | false | -| `did_not_run` | Nothing was written | true when a retry may succeed | +| `did_not_run` | This call wrote nothing. After `request_id_conflict`, an earlier request with the same `request_id` may have saved | true when a retry may succeed | `running` is the one non-terminal value. The plain result is a pure function of Flow's `transaction_outcome`, the process facts (did Flow @@ -140,15 +140,19 @@ pause, and a consistency check of the persisted report: | Flow could not be launched | `platform_error` | `did_not_run` | | the app isn't reachable, checked before anything is sent (sandbox) | `app_unreachable` | `did_not_run` | | invalid inputs, unknown workflow, runs disabled | `invalid_input`, `unknown_workflow`, `runs_disabled` | `did_not_run` | +| a used `request_id` with a different workflow or inputs | `request_id_conflict`, with `record_changed: "unknown"` | `did_not_run` (no retry) | +| a `request_id` whose five attempts each proved nothing was written | `retry_limit` | `did_not_run` (no retry) | Only `done` says the change was saved. `record_changed: "no"` and `safe_to_retry: true` appear only when the evidence proves nothing was -written. A coarse `HALTED` without a transaction outcome is uncertain, -never "nothing changed". A failed Needs Attention lookup counts as an open -pause, so the run waits for a person. `proof` (`local`, `sealed`, -`simulated`, or `none`) is separate from the outcome: a verified local run -is `done` with `proof: "local"`, never an error. A partner whose policy -requires a signed receipt checks `proof`. +written. `request_id_conflict` starts nothing, but an earlier attempt +under that `request_id` may have saved, so it reports +`record_changed: "unknown"`. A coarse `HALTED` without a transaction +outcome is uncertain, never "nothing changed". A failed Needs Attention +lookup counts as an open pause, so the run waits for a person. `proof` +(`local`, `sealed`, `simulated`, or `none`) is separate from the outcome: +a verified local run is `done` with `proof: "local"`, never an error. A +partner whose policy requires a signed receipt checks `proof`. Every sentence in `what_happened` and `next_action` is fixed copy keyed by a closed reason code. No observed screen text, recorded value, input value, @@ -164,7 +168,8 @@ the winner. The same `request_id` returns the latest attempt's result with `replayed: true`. A new attempt starts only when the latest one ended in a reason from `RETRYABLE_REASONS`, each of which proves nothing was written, and at most five times. A `request_id` reused with different inputs is -refused as `request_id_conflict`. If Flow's ledger can't be opened, local +refused as `request_id_conflict`, with `record_changed: "unknown"` because +the earlier attempt may have saved. If Flow's ledger can't be opened, local exclusive-create claim files give the same at-most-once guarantee for that runs directory. diff --git a/llms.txt b/llms.txt index e89cf69..77f6c97 100644 --- a/llms.txt +++ b/llms.txt @@ -7,7 +7,7 @@ Use this when an AI agent has decided what to enter in an app with no usable API ## The contract: three tools - `list_workflows`: each workflow's `name`, `purpose`, `done_means`, and typed `inputs` (types, enums, formats). -- `run_workflow(workflow, inputs, request_id, wait_seconds?)`: one piece of work. `request_id` is the caller's own id and is required. The same `request_id` returns the first result and never writes twice. +- `run_workflow(workflow, inputs, request_id, wait_seconds?)`: one piece of work. `request_id` is the caller's own id and is required. The same `request_id` returns the latest result and never writes twice. - `get_run(run_id, wait_seconds?)`: the result of a run that returned `outcome: running`. Every result has `outcome`, `safe_to_retry`, `what_happened` (one plain sentence), `next_action`, `proof`, `run_id`, and `request_id`. Tools publish an MCP `outputSchema` and return `structuredContent`. @@ -17,7 +17,7 @@ Every result has `outcome`, `safe_to_retry`, `what_happened` (one plain sentence - `done`: saved and checked by reading the record back. `proof` is `local` (checked on the customer's computer), `sealed` (signed receipt), or `simulated` (sandbox, no app opened). A verified local run is done; it is never reported as an error. - `needs_review`: stopped before saving; nothing was written; a person decides. Don't start the same work again. - `not_sure_if_saved`: a save may have gone through. Never retry. A person checks the record. Never tell the user nothing changed. -- `did_not_run`: nothing was written. Fix the cause and retry with the same `request_id` when `safe_to_retry` is true. +- `did_not_run`: this call wrote nothing. Fix the cause and retry with the same `request_id` when `safe_to_retry` is true. After `request_id_conflict`, an earlier request with the same `request_id` may have saved; call `get_run` with `first_run_id` to see it. Only `done` is a success. Results come from openadapt-flow's `transaction_outcome`, never from the coarse halt label alone. diff --git a/skills/openadapt-gui-write/SKILL.md b/skills/openadapt-gui-write/SKILL.md index 1fa6f15..d77dae9 100644 --- a/skills/openadapt-gui-write/SKILL.md +++ b/skills/openadapt-gui-write/SKILL.md @@ -20,9 +20,9 @@ Use this when your agent has decided what to enter and the app has no usable API | `done` | Saved and checked. | Nothing more for this request. | | `needs_review` | Stopped before saving. Nothing was written. A person decides. | Don't start the same work again. Check back with `get_run`. | | `not_sure_if_saved` | It may or may not have saved. | Never retry. A person checks the record. | -| `did_not_run` | Nothing was written. | Fix the problem in `what_happened`, then retry with the same `request_id` if `safe_to_retry` is true. | +| `did_not_run` | This call wrote nothing. After `request_id_conflict`, an earlier request with the same `request_id` may have saved. | Fix the problem in `what_happened`, then retry with the same `request_id` if `safe_to_retry` is true. | -Only outcome done means the change was saved and checked. needs_review means it stopped before saving and a person decides, so don't start the same work again. not_sure_if_saved means a person must check the record, so never retry it. did_not_run means nothing was written: fix the problem and retry with the same request_id when safe_to_retry is true. +Only outcome done means the change was saved and checked. needs_review means it stopped before saving and a person decides, so don't start the same work again. not_sure_if_saved means a person must check the record, so never retry it. did_not_run means this call wrote nothing: fix the problem and retry with the same request_id when safe_to_retry is true. After request_id_conflict, an earlier request with the same request_id may have saved, so call get_run with first_run_id to see it. `proof` says how strong the evidence behind `done` is: `local` means the record check ran on the computer that ran the workflow, `sealed` means a signed receipt exists, and `simulated` means a sandbox result that opened no app. diff --git a/src/openadapt_agent/bridge.py b/src/openadapt_agent/bridge.py index fc72545..7f7894b 100644 --- a/src/openadapt_agent/bridge.py +++ b/src/openadapt_agent/bridge.py @@ -131,10 +131,11 @@ class ToolSpec: "list_workflows, its inputs, and your own request_id for this piece of " "work. The outcome is done (saved and checked), needs_review (stopped " "before saving; a person decides), not_sure_if_saved (a person must check " - "the record; never retry), or did_not_run (nothing was written; fix it " - "and retry when safe_to_retry is true). Reuse the same request_id when " - "you retry: the same id never writes twice. If the outcome is running, " - "call get_run with the run_id." + "the record; never retry), or did_not_run (this call wrote nothing; fix " + "it and retry when safe_to_retry is true; after request_id_conflict, an " + "earlier request with the same request_id may have saved). Reuse the same " + "request_id when you retry: the same id never writes twice. If the " + "outcome is running, call get_run with the run_id." ) _RUN_WORKFLOW_USAGE = ( "run_workflow needs workflow (a name from list_workflows), inputs (an " diff --git a/src/openadapt_agent/cli.py b/src/openadapt_agent/cli.py index ba24d82..3cbbe90 100644 --- a/src/openadapt_agent/cli.py +++ b/src/openadapt_agent/cli.py @@ -405,6 +405,7 @@ def _cmd_serve(args: argparse.Namespace) -> int: return 0 if mode == "sandbox": + from openadapt_agent.copy import TUTORIAL_SERVE_COMMAND from openadapt_agent.sandbox import SandboxEngine, default_sandbox_dir runs_dir = Path(args.runs_dir).expanduser() if args.runs_dir else default_sandbox_dir() @@ -422,7 +423,7 @@ def _cmd_serve(args: argparse.Namespace) -> int: if sandbox_engine.unavailable_reason == "browser_extra_missing": note = ( " To drive the synthetic app in a hidden browser instead, " - "install the tutorial extra." + f"install the tutorial extra: {TUTORIAL_SERVE_COMMAND}" ) print( f"openadapt-agent {__version__}: sandbox mode. One workflow " diff --git a/src/openadapt_agent/contract.py b/src/openadapt_agent/contract.py index 4e87a1e..0575d4c 100644 --- a/src/openadapt_agent/contract.py +++ b/src/openadapt_agent/contract.py @@ -19,7 +19,9 @@ - ``record_changed: "no"`` and ``safe_to_retry: true`` appear only for results whose evidence proves nothing was written. Uncertain delivery (including a timeout after Flow started) is ``not_sure_if_saved`` with - ``safe_to_retry: false``. + ``safe_to_retry: false``. A ``request_id_conflict`` starts nothing, but an + earlier attempt under that ``request_id`` may have saved, so it reports + ``record_changed: "unknown"``. - A run with an open durable pause is ``needs_review``: a person holds the decision, so the caller must not start the same work again. - Every sentence is fixed copy keyed by a closed reason code. No observed screen @@ -246,7 +248,7 @@ class Reason: "the server closed. The change may or may not be saved.", _CHECK_RECORD, ), - # -- did_not_run: proven nothing written ------------------------------- + # -- did_not_run: this call wrote nothing -------------------------------- "invalid_input": Reason( "did_not_run", True, @@ -341,13 +343,16 @@ class Reason: "Don't retry this request. If the work is still needed, a person " "decides how, and a new attempt uses a new request_id.", ), + # The earlier request under this request_id may have saved, so this + # result never says nothing was written. "request_id_conflict": Reason( "did_not_run", False, - "no", + "unknown", "fix_inputs", "Didn't start because this request_id was already used for a " - "different request. Nothing new ran.", + "different request. Nothing new ran, but the earlier request may " + "have saved.", "Use a new request_id for new work. To see the earlier result, call " "get_run with first_run_id.", ), @@ -729,8 +734,10 @@ def build_result( "description": ( "done: saved and checked. needs_review: stopped before saving; a " "person decides. not_sure_if_saved: a person must check the record " - "before anything runs again; never retry. did_not_run: nothing was " - "written. running: call get_run later." + "before anything runs again; never retry. did_not_run: this call " + "wrote nothing; after request_id_conflict an earlier request " + "with the same request_id may have saved. running: call get_run " + "later." ), }, "label": {"type": "string"}, diff --git a/src/openadapt_agent/copy.py b/src/openadapt_agent/copy.py index df57aa8..4106c9b 100644 --- a/src/openadapt_agent/copy.py +++ b/src/openadapt_agent/copy.py @@ -21,8 +21,10 @@ "Only outcome done means the change was saved and checked. needs_review means " "it stopped before saving and a person decides, so don't start the same work " "again. not_sure_if_saved means a person must check the record, so never retry " - "it. did_not_run means nothing was written: fix the problem and retry with the " - "same request_id when safe_to_retry is true." + "it. did_not_run means this call wrote nothing: fix the problem and retry with " + "the same request_id when safe_to_retry is true. After request_id_conflict, an " + "earlier request with the same request_id may have saved, so call get_run with " + "first_run_id to see it." ) #: Older name for ``OUTCOME_RULES``. SKILL_HONESTY = OUTCOME_RULES @@ -38,3 +40,14 @@ ) #: Older name for ``FIRST_COMMAND``. THREE_LINE_INSTALL = FIRST_COMMAND + +#: The tutorial extra from the GitHub source. PyPI has 2.0.1, which has no +#: sandbox, so ``openadapt-agent[tutorial]`` from PyPI starts a server that +#: exits. Drop the ``@ git+...`` part once 2.0.2 is on PyPI. +TUTORIAL_REQUIREMENT = ( + "openadapt-agent[tutorial] @ git+https://github.com/OpenAdaptAI/openadapt-agent" +) +#: Starts the sandbox with the real engine. The README shows the same command. +TUTORIAL_SERVE_COMMAND = ( + f"uvx --python 3.12 --from '{TUTORIAL_REQUIREMENT}' openadapt-agent serve" +) diff --git a/src/openadapt_agent/mcp.py b/src/openadapt_agent/mcp.py index 1929d26..98b77d5 100644 --- a/src/openadapt_agent/mcp.py +++ b/src/openadapt_agent/mcp.py @@ -29,6 +29,7 @@ from mcp.server.lowlevel import Server from mcp.server.stdio import stdio_server +from openadapt_agent import __version__ from openadapt_agent.attended import ATTENDED_TOOLS from openadapt_agent.authoring import AuthoringBridge from openadapt_agent.bridge import AgentBridge, BridgeError @@ -116,11 +117,13 @@ def _server_instructions( "saved and checked; needs_review means it stopped before saving and " "a person decides, so don't start the same work again; " "not_sure_if_saved means a person must check the record, so never " - "retry it; did_not_run means nothing was written, so fix the problem " - "and retry only when safe_to_retry is true. Reuse the same request_id " - "when you retry: the same id never writes twice. If the outcome is " - "running, call get_run with the run_id. Tell the user what_happened " - "in your own words and never call anything but done a success." + "retry it; did_not_run means this call wrote nothing, so fix the " + "problem and retry only when safe_to_retry is true, and after " + "request_id_conflict an earlier request with the same request_id may " + "have saved. Reuse the same request_id when you retry: the same id " + "never writes twice. If the outcome is running, call get_run with " + "the run_id. Tell the user what_happened in your own words and " + "never call anything but done a success." ) if bridge.mode == "sandbox": text += ( @@ -253,7 +256,11 @@ def call() -> dict[str, Any]: return types.CallToolResult(content=text) if hasattr(Server, "list_tools"): - server = Server(SERVER_NAME, instructions=_server_instructions(authoring, bridge)) + server = Server( + SERVER_NAME, + version=__version__, + instructions=_server_instructions(authoring, bridge), + ) server.list_tools()(_list_tools) async def _call_v1(name: str, arguments: dict[str, Any] | None): @@ -271,6 +278,7 @@ async def _call_v2(context: Any, params: types.CallToolRequestParams): server = Server( SERVER_NAME, + version=__version__, instructions=_server_instructions(authoring, bridge), on_list_tools=_list_v2, on_call_tool=_call_v2, diff --git a/src/openadapt_agent/sandbox.py b/src/openadapt_agent/sandbox.py index 3cb0d08..a54505a 100644 --- a/src/openadapt_agent/sandbox.py +++ b/src/openadapt_agent/sandbox.py @@ -47,6 +47,7 @@ from openadapt_agent.cards import WorkflowCard from openadapt_agent.contract import parse_failed_checks, reason_for_run +from openadapt_agent.copy import TUTORIAL_REQUIREMENT, TUTORIAL_SERVE_COMMAND from openadapt_agent.service import ( CatalogEntry, EngineResult, @@ -237,7 +238,7 @@ def __init__( elif engine == "flow": raise RuntimeError( "the flow sandbox engine needs the tutorial extra: " - "pip install 'openadapt-agent[tutorial]'" + f"pip install '{TUTORIAL_REQUIREMENT}'" ) else: self.engine = "simulated" @@ -285,7 +286,7 @@ def describe(self) -> dict[str, Any]: info["engine_detail"] = ( "Results are simulated without opening the app. To drive the " "synthetic app in a hidden browser, install the tutorial extra: " - "uvx --python 3.12 --from 'openadapt-agent[tutorial]' openadapt-agent serve" + + TUTORIAL_SERVE_COMMAND ) return info diff --git a/src/openadapt_agent/skill.py b/src/openadapt_agent/skill.py index 8f52f4e..06cf297 100644 --- a/src/openadapt_agent/skill.py +++ b/src/openadapt_agent/skill.py @@ -20,7 +20,7 @@ import json import shutil from pathlib import Path -from typing import Any +from typing import Any, Optional from openadapt_agent.bundles import load_workflow_info from openadapt_agent.cards import WorkflowCard, card_for_bundle, inputs_schema @@ -28,15 +28,23 @@ __all__ = ["emit_agent_skill", "render_skill"] -_FORBIDDEN_SKILL_NAMES = frozenset({"computer-use", "computer_use", "computeruse"}) _DESCRIPTION_LIMIT = 1024 -def _skill_name(card_name: str) -> str: +def _skill_name(card_name: str, workflow_id: Optional[str]) -> str: + """Name a workflow's skill after its card, avoiding two reserved names. + + A skill named ``computer-use`` would load for every request to control the + screen, and ``SKILL_NAME`` is the general skill this package ships, which + an emitted skill in the same folder would replace. A workflow whose name + maps to either one gets ``openadapt-`` instead. + """ name = card_name.replace("_", "-").lower()[:64].strip("-") - if name in _FORBIDDEN_SKILL_NAMES or name.replace("-", "") == "computeruse": - return SKILL_NAME - return name or SKILL_NAME + if name and name.replace("-", "") != "computeruse" and name != SKILL_NAME: + return name + if not workflow_id: + raise ValueError("this workflow's skill name is reserved; pass its workflow_id") + return f"openadapt-{workflow_id}".replace("_", "-").lower()[:64].strip("-") def _input_line(name: str, schema: dict[str, Any], required: bool) -> str: @@ -69,8 +77,16 @@ def _description(card: WorkflowCard) -> str: return text[:_DESCRIPTION_LIMIT] -def render_skill(card: WorkflowCard, *, include_bundle: bool = False) -> str: - """Render SKILL.md for one workflow card. Pure: no I/O.""" +def render_skill( + card: WorkflowCard, + *, + include_bundle: bool = False, + workflow_id: Optional[str] = None, +) -> str: + """Render SKILL.md for one workflow card. Pure: no I/O. + + ``workflow_id`` names the skill when the card's name is reserved. + """ schema = inputs_schema(card) required = set(schema["required"]) inputs = "\n".join( @@ -92,7 +108,7 @@ def render_skill(card: WorkflowCard, *, include_bundle: bool = False) -> str: "operator's server. It is protected workflow data. Don't open or quote it.\n" ) return f"""--- -name: {_skill_name(card.name)} +name: {_skill_name(card.name, workflow_id)} description: {json.dumps(_description(card))} --- @@ -133,7 +149,7 @@ def render_skill(card: WorkflowCard, *, include_bundle: bool = False) -> str: | `done` | Saved and checked. | Nothing more for this request. | | `needs_review` | Stopped before saving. Nothing was written. A person decides. | Don't start the same work again. Check back with `get_run`. | | `not_sure_if_saved` | It may or may not have saved. | Never retry. A person checks the record. | -| `did_not_run` | Nothing was written. | Fix the problem in `what_happened`, then retry with the same `request_id` if `safe_to_retry` is true. | +| `did_not_run` | This call wrote nothing. After `request_id_conflict`, an earlier request with the same `request_id` may have saved. | Fix the problem in `what_happened`, then retry with the same `request_id` if `safe_to_retry` is true. | {OUTCOME_RULES} @@ -161,10 +177,11 @@ def emit_agent_skill( if not info.ok: raise ValueError("the workflow bundle could not be loaded safely") card = card_for_bundle(info, default_name=info.public_id) - skill_dir = Path(out_dir) / _skill_name(card.name) + skill_dir = Path(out_dir) / _skill_name(card.name, info.public_id) skill_dir.mkdir(parents=True, exist_ok=True) (skill_dir / "SKILL.md").write_text( - render_skill(card, include_bundle=include_bundle), encoding="utf-8" + render_skill(card, include_bundle=include_bundle, workflow_id=info.public_id), + encoding="utf-8", ) if include_bundle: shutil.copytree(bundle_dir, skill_dir / "bundle", dirs_exist_ok=True) diff --git a/src/openadapt_agent/tutorial.py b/src/openadapt_agent/tutorial.py index 2354580..c32b15c 100644 --- a/src/openadapt_agent/tutorial.py +++ b/src/openadapt_agent/tutorial.py @@ -11,6 +11,8 @@ from pathlib import Path from typing import Callable +from openadapt_agent.copy import TUTORIAL_REQUIREMENT, TUTORIAL_SERVE_COMMAND + __all__ = [ "TutorialError", "TutorialSession", @@ -127,9 +129,8 @@ def prepare_tutorial_session( except ImportError as exc: raise TutorialError( "serve --tutorial needs the browser extra. Install with " - "pip install 'openadapt-agent[tutorial]' or " - "uvx --from 'openadapt-agent[tutorial]' openadapt-agent " - "serve --allow-run" + f"pip install '{TUTORIAL_REQUIREMENT}' or start the server with " + f"{TUTORIAL_SERVE_COMMAND}" ) from exc root = Path(work_dir) diff --git a/tests/test_contract.py b/tests/test_contract.py index 358a90b..be874c2 100644 --- a/tests/test_contract.py +++ b/tests/test_contract.py @@ -195,6 +195,21 @@ def test_every_result_validates_against_the_output_schema(): jsonschema.validate(result, RUN_RESULT_SCHEMA) +def test_request_id_conflict_never_says_nothing_was_written(): + # An earlier attempt under the same request_id may have saved. + spec = REASONS["request_id_conflict"] + assert spec.outcome == "did_not_run" + assert spec.safe_to_retry is False + assert spec.record_changed == "unknown" + assert "nothing was written" not in spec.what_happened.lower() + + +def test_only_proven_no_write_results_say_nothing_was_written(): + for name, spec in REASONS.items(): + if "nothing was written" in spec.what_happened.lower(): + assert spec.record_changed == "no", name + + def test_only_done_claims_a_save_and_only_proven_cases_allow_retry(): for name, spec in REASONS.items(): if spec.record_changed == "yes": diff --git a/tests/test_distribution.py b/tests/test_distribution.py index cde77a7..c72b20f 100644 --- a/tests/test_distribution.py +++ b/tests/test_distribution.py @@ -26,6 +26,7 @@ OUTCOME_RULES, PREVIEW_COMMAND, SKILL_WHEN_TO_USE, + TUTORIAL_SERVE_COMMAND, ) REPO_ROOT = Path(__file__).resolve().parents[1] @@ -34,6 +35,7 @@ LLMS_TXT = REPO_ROOT / "llms.txt" PYPROJECT = REPO_ROOT / "pyproject.toml" README = REPO_ROOT / "README.md" +DESIGN = REPO_ROOT / "docs" / "DESIGN.md" RELEASE_WORKFLOW = REPO_ROOT / ".github" / "workflows" / "release.yml" WORKFLOWS_DIR = REPO_ROOT / ".github" / "workflows" UV_LOCK = REPO_ROOT / "uv.lock" @@ -331,6 +333,7 @@ def test_retired_unsafe_claims_never_come_back() -> None: encoding="utf-8" ), "server.json": SERVER_JSON.read_text(encoding="utf-8"), + "DESIGN.md": DESIGN.read_text(encoding="utf-8"), } for name, text in surfaces.items(): for retired in ( @@ -338,8 +341,21 @@ def test_retired_unsafe_claims_never_come_back() -> None: "record did not change", "Production success without a Seal is failure", "Unsigned production success is failure", + # did_not_run includes request_id_conflict, where an earlier + # request under the same request_id may have saved. + "did_not_run means nothing was written", + "`did_not_run`: nothing was written", + "| `did_not_run` | Nothing was written", ): assert retired not in text, (name, retired) + assert "nothing was written" not in OUTCOME_RULES.split("did_not_run", 1)[1] + + +def test_tutorial_command_installs_a_version_with_the_sandbox() -> None: + """Until 2.0.2 is on PyPI, the tutorial extra comes from GitHub.""" + readme = README.read_text(encoding="utf-8") + assert TUTORIAL_SERVE_COMMAND in readme + assert "--from 'openadapt-agent[tutorial]' " not in readme def _without_code(text: str) -> str: diff --git a/tests/test_mcp_server.py b/tests/test_mcp_server.py index eb1d72c..f3deb77 100644 --- a/tests/test_mcp_server.py +++ b/tests/test_mcp_server.py @@ -12,6 +12,7 @@ import mcp.types as types import pytest +from openadapt_agent import __version__ from openadapt_agent.attended import ATTENDED_TOOLS from openadapt_agent.bridge import AgentBridge from openadapt_agent.mcp import build_server @@ -61,6 +62,12 @@ async def list_tools(): assert list_tool.get("_meta") is None +def test_initialize_reports_the_package_version(bundles_root, runner_config): + """serverInfo.version comes from these options; the SDK default is empty.""" + server = build_server(AgentBridge(bundles_root, runner_config)) + assert server.create_initialization_options().server_version == __version__ + + def test_server_read_only_when_run_not_allowed(bundles_root, runner_config, mcp_client): bridge = AgentBridge(bundles_root, runner_config, allow_run=False) server = build_server(bridge) @@ -304,6 +311,7 @@ async def probe(): initialized, names, result, refused = anyio.run(probe) assert initialized["serverInfo"]["name"] == "openadapt-agent" + assert initialized["serverInfo"]["version"] == __version__ assert "list_workflows" in names assert result["isError"] is False assert json.loads(result["content"][0]["text"])["run_tools_enabled"] is False diff --git a/tests/test_sandbox.py b/tests/test_sandbox.py index a36e861..3caa8d4 100644 --- a/tests/test_sandbox.py +++ b/tests/test_sandbox.py @@ -12,6 +12,7 @@ from openadapt_agent.bridge import AgentBridge from openadapt_agent.contract import RUN_RESULT_SCHEMA +from openadapt_agent.copy import TUTORIAL_SERVE_COMMAND from openadapt_agent.runner import RunnerConfig from openadapt_agent.sandbox import CASES, SANDBOX_WORKFLOW, SandboxEngine @@ -68,6 +69,15 @@ def test_each_case_returns_its_outcome(sandbox_bridge, case): assert "did not change" not in result["what_happened"] +def test_simulated_engine_hint_installs_a_version_with_the_sandbox(sandbox_bridge): + # PyPI 2.0.1 has no sandbox, so a hint that installs the tutorial extra + # from PyPI starts a server that exits. Point at the GitHub source. + detail = sandbox_bridge.dispatch("list_workflows", {})["sandbox"]["engine_detail"] + assert TUTORIAL_SERVE_COMMAND in detail + assert "git+https://github.com/OpenAdaptAI/openadapt-agent" in detail + assert "--from 'openadapt-agent[tutorial]' " not in detail + + def test_app_offline_says_the_app_is_unreachable(sandbox_bridge): result = run(sandbox_bridge, case="app_offline", request_id="demo-offline") assert result["reason"] == "app_unreachable" @@ -94,6 +104,8 @@ def test_same_request_id_replays(sandbox_bridge): assert again["replayed"] is True changed = run(sandbox_bridge, case="normal") assert changed["reason"] == "request_id_conflict" + assert changed["record_changed"] == "unknown" + assert "nothing was written" not in changed["what_happened"].lower() def test_get_run_returns_the_stored_result(sandbox_bridge): diff --git a/tests/test_service.py b/tests/test_service.py index 48dd152..3e7acb1 100644 --- a/tests/test_service.py +++ b/tests/test_service.py @@ -157,6 +157,9 @@ def test_reused_request_id_with_new_inputs_is_refused( assert conflict["reason"] == "request_id_conflict" assert conflict["first_run_id"] == first["run_id"] assert conflict["safe_to_retry"] is False + # The first request saved, so the conflict can't say nothing was written. + assert conflict["record_changed"] == "unknown" + assert "nothing was written" not in conflict["what_happened"].lower() def test_slow_run_returns_running_then_get_run_waits_for_it( diff --git a/tests/test_skill.py b/tests/test_skill.py index 6a79883..31d4a48 100644 --- a/tests/test_skill.py +++ b/tests/test_skill.py @@ -5,10 +5,15 @@ import json from pathlib import Path -from openadapt_agent.cards import CARD_FILENAME +import pytest + +from openadapt_agent.bundles import load_workflow_info +from openadapt_agent.cards import CARD_FILENAME, card_for_bundle from openadapt_agent.cli import main -from openadapt_agent.copy import OUTCOME_RULES -from openadapt_agent.skill import emit_agent_skill +from openadapt_agent.copy import OUTCOME_RULES, SKILL_NAME +from openadapt_agent.skill import emit_agent_skill, render_skill + +REPO_ROOT = Path(__file__).resolve().parents[1] SECRET_VALUES = ("Follow-up in 2 weeks", "Open the patient chart", "Type the triage note") @@ -116,3 +121,44 @@ def test_skill_folder_is_valid_markdown_with_frontmatter(bundle_dir, tmp_path): assert len(fields["name"]) <= 64 assert len(json.loads(fields["description"])) <= 1024 assert Path(tmp_path / "s").is_dir() + + +def _name_card(bundle_dir, name): + (bundle_dir / CARD_FILENAME).write_text( + json.dumps({"name": name, "purpose": "Add a triage note to a chart."}) + ) + + +@pytest.mark.parametrize("reserved", ["computer_use", "openadapt_gui_write"]) +def test_reserved_names_fall_back_to_the_workflow_id(bundle_dir, tmp_path, reserved): + _name_card(bundle_dir, reserved) + workflow_id = load_workflow_info(bundle_dir).public_id + skill_dir = emit_agent_skill(bundle_dir, tmp_path / "skills") + expected = "openadapt-" + workflow_id.replace("_", "-") + assert skill_dir.name == expected + text = (skill_dir / "SKILL.md").read_text() + assert frontmatter(text)["name"] == expected + assert "name: computer-use" not in text.lower() + assert f"name: {SKILL_NAME}\n" not in text + + +def test_emitting_never_replaces_the_shipped_general_skill(bundle_dir, tmp_path): + shipped = (REPO_ROOT / "skills" / SKILL_NAME / "SKILL.md").read_text() + skills = tmp_path / "skills" + (skills / SKILL_NAME).mkdir(parents=True) + (skills / SKILL_NAME / "SKILL.md").write_text(shipped) + for name in ("computer_use", "openadapt_gui_write"): + _name_card(bundle_dir, name) + skill_dir = emit_agent_skill(bundle_dir, skills, include_bundle=True) + assert skill_dir.name != SKILL_NAME + assert (skills / SKILL_NAME / "SKILL.md").read_text() == shipped + assert not (skills / SKILL_NAME / "bundle").exists() + + +def test_render_skill_needs_a_workflow_id_for_a_reserved_name(bundle_dir): + _name_card(bundle_dir, "computer_use") + info = load_workflow_info(bundle_dir) + card = card_for_bundle(info, default_name=info.public_id) + with pytest.raises(ValueError): + render_skill(card) + assert "name: openadapt-workflow-" in render_skill(card, workflow_id=info.public_id)