Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
31 changes: 29 additions & 2 deletions docs/explanation/limits.md
Original file line number Diff line number Diff line change
Expand Up @@ -40,14 +40,41 @@ description: >-

- **RDP and Citrix.** With only pixels, identifiers with confusable
characters can't always be read with confidence, so runs stop more than in
a browser. Teach the specific case, or use a surface with structure where
the app offers one. Don't lower the check.
a browser. On pixel-only screens the right-record check stops whenever the
right patient's own record number has look-alike characters, such as `O`
and `0`. On our test corpora that was 47 to 61 in 100 correct-patient
checks, re-measured 17 October 2026
([identity evidence](https://github.com/OpenAdaptAI/OpenAdapt/blob/main/packages/flow/docs/validation/IDENTITY_EVIDENCE.md)). Those are
extra stops, not wrong entries. A confirmed export of the system's record
numbers (`runtime.known_ids`, openadapt-flow 2.2.0) lifts a stop when
exactly one listed number fits; on a rendered record list, stops on the
right row fell from 176 to 3 in 280 clicks whose neighboring row wasn't a
look-alike twin ([results](https://github.com/OpenAdaptAI/OpenAdapt/blob/main/packages/flow/benchmark/known_ids/KNOWN_IDS.md)). Teach the specific case, or use a surface with structure where the
app offers one. Don't lower the check.
- **First runs of a new recording.** Expect `lint` warnings and
`clinical-write` refusals until record checks and right-record checks are
complete.
- **Screens that change between finding a target and acting**, such as a
late notification. The step stops rather than click a stale target.

## Limits by surface

Each row is what's on record for that surface and what that evidence doesn't
show. A given application on a given surface still needs its own
qualification.

| Surface | On record | What it doesn't show |
|---|---|---|
| Browser | OpenEMR referral entry, 14 to 15 Oct 2026: of 63 new-patient referrals, 62 saved and read back 12 of 12 fields; 9 of 9 possible duplicates with no new chart, 0 model calls ([run data](https://github.com/OpenAdaptAI/OpenAdapt/blob/main/packages/flow/benchmark/openemr_referral_timing/README.md)) | One workflow on a local test copy with synthetic patients. The read-back was our test setup's, so the engine labeled the runs finished, not checked. No commercial EMR tested. |
| Windows | 3 of 3 saves in our WinForms app confirmed in SQLite; 3 of 3 refusals each for stale and ambiguous targets; 0 model calls ([report](https://github.com/OpenAdaptAI/OpenAdapt/blob/main/packages/flow/benchmark/windows_uia/results.json)) | Our own app on one Windows 11 ARM VM snapshot. RDP and Citrix recordings don't use local UI Automation. |
| macOS | 3 of 3 TextEdit saves matched byte for byte; refused an ambiguous two-window target without changing either file ([adjudication](https://github.com/OpenAdaptAI/OpenAdapt/blob/main/packages/flow/benchmark/macos_native/textedit_counted_3plus1_b1b61a5_20260717.adjudication.json)) | One macOS 15.7.3 host and one app. The original report reads failed because of cleanup warnings; the adjudication accepts the saves and the refusal. |
| Linux (X11) | In required CI on Ubuntu 24.04: a GTK3 app, 3 confirmed saves, 3 ambiguous-target and 3 stale-target refusals | Our own app on Xvfb. No Wayland, no third-party apps. |
| RDP | Windows 11 VM over Aardwolf: 3 of 3 file writes confirmed by a separate guest check ([report](https://github.com/OpenAdaptAI/OpenAdapt/blob/main/packages/flow/benchmark/rdp/results_82a658a_20260718.sanitized.json)). Record, compile and replay over FreeRDP: 3 of 3 saves, 3 of 3 stops under drift ([report](https://github.com/OpenAdaptAI/OpenAdapt/blob/main/packages/flow/benchmark/rdp_ladder/results.json)) | The FreeRDP run used a synthetic Linux app, not a Windows app, and drift wasn't captured over a WAN. Pixel-only, so the false-stop rate above applies. |
| Citrix | 3 confirmed saves and 3 stops under severe drift on a stand-in window with no DOM; 0 model calls ([report](https://github.com/OpenAdaptAI/OpenAdapt/blob/main/packages/flow/benchmark/citrix_workspace/results.json)) | No run over a real Citrix ICA/HDX connection is on record. Each Citrix deployment is qualified in its own environment. Pixel-only, so the false-stop rate above applies. |

The full failure inventory, with evidence for each entry, is in
[`packages/flow/docs/LIMITS.md`](https://github.com/OpenAdaptAI/OpenAdapt/blob/main/packages/flow/docs/LIMITS.md).

## Surfaces and lanes still being proven

- Citrix has no counted run over real ICA/HDX on record yet.
Expand Down
4 changes: 3 additions & 1 deletion docs/explanation/verification.md
Original file line number Diff line number Diff line change
Expand Up @@ -67,7 +67,9 @@ with an extra write to another table. Three checks judged the same runs:

Each check also stopped 9 of the 18 runs that saved correctly, because the
reply to the save never reached the client. A record check catches what its
read can see, so it should cover every table a save can change.
read can see, so it should cover every table a save can change. These
were faults we wrote and injected into our own engine: a coverage test over a
fixed list, not a real-world error rate.
[Method and per-fault results](https://github.com/OpenAdaptAI/OpenAdapt/blob/main/packages/flow/benchmark/effect_e2e/EFFECT_E2E.md).

## How a save is approved
Expand Down
Loading