Smalltalk Product And Technical Specification

Last updated: 2026-07-11

Implementation baseline: the live checkout and current dirty working tree through P6.09, including the July 11 legacy-schema migration repair and live Codex output audit. ab37fa0b remains the last reviewed orientation commit, not the implementation boundary or a reset target. The repository is on a case-insensitive filesystem; PRODUCT.md is the canonical tracked filename even when tools display it as product.md.

This document describes the current Smalltalk product as implemented in this repository. It is written for another engineer or LLM that needs to understand what the product is, how it works, which parts are active, which parts are diagnostic, and where the current architecture still leaks older recorder behavior.

Smalltalk is a desktop-first, local-first continuation product built with Tauri, Rust, React, SQLite, macOS native capture APIs, Accessibility, OCR, and optional OpenAI calls. Its primary user-facing primitive is Continue: a single evidence-backed answer that explains what the user was doing, where they were doing it, what state was left behind, what should happen next, and whether an exact safe return target exists. The latest factual screen and the recommended return target remain separate facts.

The active product lane is the native desktop app in the repository root. The older WXT browser extension remains in browser-extension/, but it is not the MVP path and should not be revived unless a task explicitly asks for browser-extension work.

Changes Since The Previous Product Snapshot

The previous product snapshot was centered on the committed P1-P4 Continue hardening plus the then-current P5 activity-recap working tree. The live checkout and current P6 working tree materially extend that baseline:

Capture is explicitly sparse and event-driven. Native app, focus, Accessibility, click, key-category, scroll, and clipboard signals are stored cheaply; screenshots, AX trees, OCR, window graphs, and normalized content are stored only for accepted heavy frames.
Long-running local memory now has concrete pressure controls: event coalescing, native scroll/AX throttling, 4-second important and 45-second low-value heavy-capture intervals, a 24-frame rolling screenshot budget for low-value captures, three unchanged heavy frames per surface, a 512 MB snapshot pressure gate, image-plus-content deduplication, Smalltalk self-capture suppression, and cleanup caps for low-value frames and events.
Text attribution now separates active owned text from background or display-only OCR through frame_text_resolutions. Mixed/background text can remain evidence without being trusted as primary task text.
Continue gained semantic moments, boundary revisions, app-activity segments, open loops, workstream state snapshots/edges, memory cells/edges, pairwise preferences, ranking priors, and decision-open telemetry.
Current focus is no longer equivalent to the latest screenshot. resolve_current_surface fuses frames, events, artifact observations, app contexts, window state, and typing activity; the evidence-freshness ledger then compares that current surface with the selected target.
Fresh non-openable current work can outrank a stale openable target. active_current_work_unresolved is a factual first-class result, while public return_target and resume_work_target stay null when the exact safe target is missing.
Repeated negative feedback is enforced before ranking, in alternatives, in the model candidate pack, during validation, during cache reuse, and again at strict open time. Fresh reconfirming local evidence is required before a suppressed target can return.
Search, docs, messages, diagnostics, terminal support output, and other branches are evidence-only by default. Explicit local branch-promotion state is required before a support branch can become a public return target.
Weak native/editor/terminal surfaces now have bounded enrichment attempts, normalized surface snapshots, stable hashed identities, evidence-quality grading, missing-evidence labels, and truthful thin-state handling.
Continue audits are opt-in per explicit Continue action, asynchronous, proof-first, and lean by default. Startup/background refreshes do not create bundles; full SQLite/table/frame archives require an explicit full-raw mode.
The floating island is now a Continue consumer instead of a legacy resume bypass. It renders the typed IslandContinueState, opens only through a persisted continue_decision_id, records source-aware feedback/open telemetry, and fails closed when the main Continue policy would suppress the target.
SQLite access is hardened for long-running capture and concurrent status/Continue work: writable connections use WAL and synchronous=NORMAL, all connections use a 30-second busy timeout, read-only polling uses read-only connections, and get_continue_decision is serialized by a process-level lock.
P5 adds smalltalk.activity_recap.v1, an explanatory activity-memory contract on top of the existing decision engine. It distinguishes the inferred primary activity from target selection and carries separate activity and target confidence.
P5 builds a bounded, privacy-filtered input pack from current-surface facts, activity segments, actions, semantic moments, open loops, workstream state, branch contexts, surface snapshots, support evidence, and safe memory cells. It does not fall back to sending broad raw history.
P5 stitches those facts into a compact activity timeline, infers grounded work labels and where the work happened, distinguishes support/detours/interruptions from primary work, and synthesizes the last meaningful, unfinished, blocked, complete, idle, or unclear state plus a safe next action.
Optional recap model phrasing is disabled by default through activity_recap_model_enabled. When enabled, it can rewrite only a bounded local fact pack; local validation rejects unsupported claims and falls back to the deterministic recap.
Recap validity participates in decision cache identity through a recap watermark and policy fingerprint. Cache hits reuse the exact persisted recap, meaningful evidence or feedback invalidates it, and recap memory is explicitly prevented from influencing ranking, target eligibility, openability, or strict opening.
React and the native island now present the compact activity memory: what the work was, where, recent context, state left behind, next action, safe target, separate activity/target confidence, and missing evidence.
P6.01 adds the versioned privacy-safe accuracy-fixture contract, frozen evaluation policy, first-divergence replay, known-failure milestones, default-denied locked-holdout access, privacy lint, deterministic replay checks, and a machine-readable accuracy report.
P6.02 persists ordered evidence spans derived from AX, OCR, content units, and bounded fallback text. It records source ownership, pane/order geometry, region role, conversational role, confidence, reason codes, and selected/rejected salient user, agent, and prior-boundary evidence.
P6.03 persists a first-class current task turn with separate execution state, current actor, waiting-on state, relation to prior, evidence links, revision, and task-turn-scoped actions/semantic deltas. Task-turn changes participate in decision cache identity.
P6.04 evaluates feedback through explicit provenance, task/session/workstream/target scope, freshness/decay, allowed effects, and per-target polarity. Stale inferred navigation cannot promote a support branch.
P6.05 adds task-turn/workstream membership, semantic eligibility, cross-layer consistency, and a strict direct-target policy that keeps frame previews and support surfaces out of public return targets.
P6.06 adds an evidence-linked confidence vector and bounded observe-before-decide outcomes for success, no change, timeout, failure, privacy block, and stale result. A strong surface cannot raise an unsupported task or target claim.
P6.07 rebases deterministic recap and optional model phrasing on a task-truth pack with claim evidence, confidence caps, workstream consistency, and target policy. Semantic model failures fall back locally rather than overriding task identity.
P6.08 adds typed target truth, evidence preview, the interruption-recovery answer, React presentation helpers, native-island parity, and strict decision-id-only opening. A known task may be useful even when no direct target exists.
P6.09 adds explicit release-level corpus, calibration, privacy, performance, model-parity, replay, and manual-QA gates. release_gate.passed is distinct from the narrower phase milestone result.
The legacy SQLite migration now adds continue_task_actions.task_turn_id before creating idx_continue_task_actions_task_turn, allowing pre-P6 databases to upgrade instead of failing schema initialization.

Commit inventory reviewed for this update:

Commit	Date	Subject
`2e48278d`	2026-07-05	Rework Continue surface and continuation flow
`21843795`	2026-07-07	Refactor continuation flow and evidence rendering
`978dfafd`	2026-07-07	Refactor continue flow and evidence handling
`2273684d`	2026-07-07	Refactor continue evidence flow and UI state handling
`4850dfdf`	2026-07-07	Refine continue evidence recovery and candidate scoring
`9cf0e8b5`	2026-07-07	Refactor continue workflow and remove legacy resume code
`58fb77c7`	2026-07-08	Refactor continue flow and evidence handling
`2196cdc5`	2026-07-08	Refactor continuation flow and evidence handling
`abeb3622`	2026-07-08	Refine continuation flow and evidence handling
`b0e4d0a3`	2026-07-08	Refine continue flow and evidence scoring
`ab37fa0b`	2026-07-10	Refactor Continue decision flow and island audit checks

Because those commit subjects are broad, the rest of this document describes the current code paths and persisted contracts rather than trying to infer behavior from commit titles alone.

Product Doctrine

Smalltalk is continuation-first, not session-recorder-first.

The first product screen should answer:

What is the factual current focus?
What was the user actually trying to do?
In which app, page, conversation, file, or surface were they doing it?
Which recent surfaces were primary work, support, detours, or interruptions?
What meaningful, unfinished, blocked, complete, or unclear state was left behind?
What should the user do next?
Is there an actionable, safely openable return target?
What evidence supports each claim?
What evidence is missing or thin?

Sessions, screenshots, timelines, raw events, frame inspectors, cloud resume bundles, native resume cards, search, evals, candidate score components, artifact-role tables, and raw database ids are support infrastructure. They are useful for evidence inspection and developer diagnostics, but they are not the default product.

The product must keep these concepts separate:

Concept	Meaning	Should it become the return target by default?
`current_focus`	The latest factual screen or artifact observed locally.	No. It may be a distraction, support page, diagnostic surface, or current app.
`current_activity`	A local read of what appears to be happening now.	No. It explains current behavior only.
`activity_recap`	Evidence-backed explanation of the primary work, where it happened, detours/support, state left behind, next action, and uncertainty.	No. It explains the decision and cannot create target eligibility.
`selected_workstream`	The durable cluster of actions and artifacts Smalltalk thinks the user was working on.	Sometimes. It is the context for the decision.
`return_target`	The artifact Smalltalk thinks the user should go back to.	Yes, when evidence quality is sufficient.
`resume_work_target`	The actionable target inside the workstream, kept separate from support or branch evidence.	Yes. This is the preferred product target when present.
`branch/support surface`	Search results, docs, terminal output, messages, or other evidence used while doing work.	No, unless local evidence says the branch itself is the unfinished task.

Smalltalk must not invent artifacts, URLs, file paths, user intent, or next actions. If the evidence is thin, the product should say that evidence is thin and show inspectable anchors.

Smalltalk must not send broad raw history to a model and ask the model to invent intent. Model calls, where used, must be bounded to local candidate ids, evidence-backed, and locally validated.

Smalltalk must not store raw typed characters or full clipboard text. Keyboard and clipboard evidence are represented as categories, counts, hashes, and metadata.

Repository Shape

Path	Role
`src/`	React/Vite frontend for the Tauri desktop app.
`src/App.tsx`	Main desktop UI, Continue card, diagnostics, evidence inspector, correction controls, local memory controls.
`src/App.css`	Desktop shell, fixed top bar, scroll containment, Continue card, diagnostics, workstream and inspector styling.
`src-tauri/`	Rust/Tauri backend, command registration, capture runtime, SQLite store, Continue engine, macOS island integration.
`src-tauri/src/capture.rs`	Active runtime facade for capture, storage, search, safe exports, cloud resume, local memory diagnostics, cleanup, and Tauri command wrappers.
`src-tauri/src/continuation.rs`	Native Continue semantic memory, rebuild layers, scoring, decisions, feedback, breadcrumbs, eval, and default bounded micro-inference.
`src-tauri/src/continuation/activity_recap*.rs`	P5 recap contract, bounded inputs, segment stitching, grounded work labels, detour/branch recap, last-state synthesis, optional model phrasing, validation, cache/memory integration, and deterministic tests.
`src-tauri/src/continuation/task_turn_evidence.rs`	P6 ordered AX/OCR/content evidence, pane/order resolution, region and conversational roles, salient latest-turn selection, privacy-safe samples, and audit persistence.
`src-tauri/src/continuation/task_turn.rs`	P6 current-task-turn persistence, lifecycle axes, prior-turn relations, task-scoped action finalization, workstream membership, revisions, and cache markers.
`src-tauri/src/continuation/{feedback_policy,semantic_consistency,confidence,activity_recap_truth}.rs`	P6 feedback applicability, semantic-center and target policy, split confidence/probe behavior, and the canonical recap truth/guard pack.
`src-tauri/src/continuation/{accuracy_fixture,accuracy_eval}.rs` and `src-tauri/src/bin/continue_accuracy_eval.rs`	Privacy-safe P6 full-pipeline replay, metrics/calibration, deterministic/model checks, release verdict, report writer, and CLI.
`src-tauri/src/lib.rs`	Tauri builder and command registration.
`src-tauri/src/session_island.rs`	macOS floating-island Continue gateway, typed state contract, freshness memory, source-aware actions/feedback, and no-bypass audit hooks.
`src-tauri/src/session_island/`	Island contract/gateway modules and tests extracted from the bridge.
`src-tauri/macos/SessionIslandPanel.swift`	Native macOS panel UI.
`src-tauri/scripts/`	Swift helper scripts for Accessibility, OCR, window capture, native event observation, and ScreenCaptureKit support.
`src-tauri/src/capture_core/`	Newer modular capture-core code for event governance, quality, privacy, extraction, store behavior, episode policy, browser adapters, and resume dossier limits. The active facade remains `capture.rs`.
`docs/`	Technical docs, audits, architecture notes, QA notes, and product rebuild notes.
`browser-extension/`	Older browser-extension prototype. Not the active MVP lane.
`resume_query_exports/`	Generated stop-time resume-query bundles. Treat as generated unless explicitly asked to inspect.
`continue_outputs/`	Generated full Continue audit folders. Folder names start with the capture session, for example `session-001-session-id__continue-<timestamp>__normal__<decision>`. Private/debug only; do not commit.
`cloud_resume_exports/`, `output/`, `target/`, local snapshot folders	Generated artifacts. Do not commit.

Build And Verification Commands

Use these commands from the repository root unless noted otherwise:

npm install
npm run dev
npm run tauri dev
npm run build
cd src-tauri && cargo check
cd src-tauri && cargo test

The normal local app path is:

npm run tauri dev

The frontend-only Vite server is useful for UI work but does not exercise native capture:

npm run dev

For Rust backend changes, at least run:

cd src-tauri && cargo check

For deterministic Continue, parsing, storage, cleanup, and scoring changes, run the relevant Rust tests:

cd src-tauri && cargo test

Current Product Surface

The visible app is now named Smalltalk Continue.

The top bar has:

Brand block: Smalltalk and Smalltalk Continue.
Status pills for local memory, evidence age, and Continue freshness.
Primary Continue button.
Secondary Memory menu.

The Memory menu contains:

Start local memory
Pause local memory
Capture evidence now
Delete local memory

This is intentional: local capture is necessary infrastructure, but it should not be the primary product action.

The main first screen is the ContinueDecisionCard. Depending on evidence state, it shows:

No-evidence state.
Local-memory-active state.
Continue decision state.
Fresh openable enriched work state.
Fresh enriched-but-not-openable state.
Truthful thin current-work state.
Older context with fresher thin current work.
No-clear-continuation state.
You were: the supported task and work object.
State: last meaningful progress plus unfinished, waiting, or blocked state.
Next: one supported action when available; it is omitted when unsupported.
Where: app plus document, thread, or page identity at the supported precision.
Direct return target, inspectable evidence preview, or Exact location unavailable.
Answer freshness independently from task or target confidence.
Primary Continue here action.
Inspect evidence action.
Two bounded task choices only when two verified hypotheses are genuinely close.
Not right feedback scoped to the exact TaskSnapshot revision and affected field or hypothesis.

Candidate, workstream, detour, provenance, confidence, and evidence-strength details are diagnostics under Why this answer?; they are not first-screen content.

The product card always enables candidate-bounded target inference. It enables recap model phrasing only for an explicit manual Continue/Refresh/Rebuild action; startup and background refreshes keep recap synthesis local:

await invoke("get_continue_decision", {
  input: {
    mode: "normal",
    rebuild_layers: false,
    micro_inference_enabled: true,
    activity_recap_model_enabled: trigger === "manual",
    max_candidates_for_model: 5,
    audit_output_enabled: options.writeAudit === true
  }
});

The UI currently treats decision-source provenance as product-visible state. A cloud_micro_inference result with a real response_id is shown as AI-assisted; a failed or unavailable candidate-model path is shown as Local fallback; local scorer-only output is shown as Local only. This label describes the bounded candidate-routing path, not necessarily the wording shown on the card. The candidate model may validly return need_more_evidence, while activity_recap.generated_by independently remains local or fallback. Therefore the current AI-assisted badge does not prove that a model selected the task or wrote the displayed recap. This is a known provenance-presentation defect, documented in the P6 limitations below.

Manual Continue/Refresh/Rebuild requests enable optional recap-model phrasing. Startup and background requests keep recap synthesis local. React and the native island now apply a quality-dominant adoption policy: a background result may replace a stronger manual result only when its evidence is causally newer and it does not downgrade task identity, revision, confidence, supported task/state/next/where coverage, target safety, or wording provenance. Rejected background results are retained as bounded diagnostics instead of replacing the displayed answer.

The UI also filters user-facing handoff copy. If a backend or model handoff leaks internal ids or implementation labels such as candidate ids, workstream ids, artifact ids, frame fallback text, or target-metadata placeholders, the card falls back to honest thin-evidence copy instead of displaying those internals as product language.

The UI opens the selected target through:

await invoke("open_resume_point", {
  input: {
    continue_decision_id: continueDecision.decision_id,
    target_artifact_id: resumeTarget?.artifact_id || null,
    source: "desktop_continue_card",
    strict_continue_target: true
  }
});

Continue here is rendered only for openable_return_target. Fresh but non-openable work uses inspect-first copy such as Most recent work seen, Fresh current work, Exact target missing, or No safe return target yet; it does not quietly fall back to an old openable frame or generic Continue here.

The UI records explicit correction feedback through:

await invoke("record_continue_feedback", {
  input: {
    decision_id,
    selected_candidate_id,
    workstream_id,
    target_artifact_id,
    corrected_artifact_id,
    feedback_kind,
    note,
    source: "desktop_ui"
  }
});

Supported explicit feedback kinds are:

accepted
rejected
ignored
corrected
artifact_only_evidence
ignored_workstream
user_next_step_note

The frontend also has a developer diagnostics <details> panel. It contains workstream lists, breadcrumb notes, workstream detail, local memory storage diagnostics, cleanup controls, search, capture health, Continue eval, frame timeline, raw event stream, screenshot inspector, overlays, verification drawer, and raw path/context tabs. These are diagnostics, not the primary product.

Current Product Failure Boundary

The backend now has a real Continue/workstream/task-turn architecture, bounded recap truth, and target-safe answer composition. In the covered deterministic cases it can explain an activity while refusing to invent an exact page or target. The live Codex audit establishes the harder boundary: if pane, speaker, and current-user evidence resolution fail, later semantic consistency, model validation, and direct-target safety can prevent a fabricated open but cannot recover the user's real task. The product may then produce an honest target-null answer whose task summary is still irrelevant or wrong.

The visible product can still feel like an old recorder/debug dashboard when diagnostics are open. This is not just visual polish. It is a product-boundary issue.

The current repo has seven overlapping paths:

Path	Current state
React Continue shell	Primary screen is Continue-first, but diagnostics still expose many internal layers.
Continue backend	Real layered semantic memory and scoring engine.
P5 activity memory	Evidence-backed explanatory recap over the decision; it cannot override target policy or compensate for missing page/file identity.
P6 task-turn truth	Integrated ordered evidence, current task turn, semantic consistency, split confidence, target truth, and answer contract; deterministic coverage passes, but live Codex speaker recovery is not reliable enough for release.
P6 release evaluator	Produces a machine-readable longitudinal verdict; the release gate is currently closed because corpus, holdout, calibration, performance, and native QA requirements remain incomplete.
Capture/evidence backend	Operational evidence substrate with lightweight-first signals and sparse, budgeted heavy frames.
Floating island	Typed Continue-first consumer using the same backend decision and strict-open policy as the main card; legacy session/cloud routes are diagnostic-only.

The correct product direction is to make Continue the only first-screen answer and keep diagnostic internals behind secondary surfaces.

Local Storage

The live native capture store is under the Tauri app-data directory:

~/Library/Application Support/com.smalltalk.app/capture/
  smalltalk-capture.sqlite
  snapshots/
  helpers/
  safe-ai-exports/

The frontend receives exact live paths from capture_status:

data_dir
database_path

Screenshot assets are partitioned by day under snapshots/<day>/. An accepted heavy capture can write a full-display JPEG named <timestamp>_full.jpg and, when a window id is available, an active-window JPEG named <timestamp>_window.jpg. The SQLite row stores the asset paths and capture provenance; the image bytes are not embedded in SQLite.

SQLite sidecars can exist beside the main database while WAL is active:

smalltalk-capture.sqlite
smalltalk-capture.sqlite-wal
smalltalk-capture.sqlite-shm

The database is the durable local evidence and semantic-memory store across capture sessions. A one-hour session does not produce a separate database or an hour-long video. It appends lightweight event rows and a bounded number of sparse heavy frames to the same database, all linked to a capture_sessions.id.

Stop-time resume-query bundles are separate generated artifacts under the repo root:

/Users/bhaskarpandit/Documents/smalltalk/resume_query_exports/session-<sequence>-resume-query-<timestamp>-<suffix>/
  resume-query-bundle.json
  images/

Developer reset can also clear generated repo debug output:

output/
resume_query_exports/
continue_outputs/

Generated capture data, SQLite files, screenshots, safe exports, resume-query bundles, and Continue output audits must not be committed.

continue_outputs/ is not a live mirror of the database. It is written only when an explicit Continue action asks for an audit. The default bundle is a compact proof package built in a sibling .building directory and atomically renamed to its final session-readable folder after canonical proof files are complete. Background/startup decisions use audit_output_enabled: false and do not create output folders.

When debugging what the running app actually decided, first use the database_path returned by capture_status. On the default macOS bundle it is usually ~/Library/Application Support/com.smalltalk.app/capture/smalltalk-capture.sqlite. Inspect it independently of continue_outputs/, preferably read-only:

sqlite3 -readonly "$HOME/Library/Application Support/com.smalltalk.app/capture/smalltalk-capture.sqlite" \
  "SELECT id, datetime(requested_at_ms / 1000, 'unixepoch', 'localtime'), source, response_id, validation_status, current_task_turn_id, activity_recap_model_requested, micro_inference_result_kind FROM continue_decisions ORDER BY requested_at_ms DESC LIMIT 20;"

The recap wording provenance is stored inside activity_recap_json; it is not equivalent to the decision source:

sqlite3 -readonly "$HOME/Library/Application Support/com.smalltalk.app/capture/smalltalk-capture.sqlite" \
  "SELECT id, json_extract(activity_recap_json, '$.generated_by'), json_extract(activity_recap_json, '$.validation_status'), json_extract(activity_recap_json, '$.primary_work_summary') FROM continue_decisions ORDER BY requested_at_ms DESC LIMIT 20;"

Data Model Overview

The core local database is smalltalk-capture.sqlite.

Important substrate tables:

Table	Purpose
`capture_sessions`	Session id, sequence, start/stop timestamps, status, export path, and per-session counts.
`frames`	Core captured evidence rows: app, window, URL/path, text, screenshot paths, hashes, trigger, privacy, session, ScreenCaptureKit metadata.
`frames_fts`	FTS5 index over frame text and metadata.
`ocr_text`	One row per frame for joined OCR text and raw OCR JSON.
`ocr_spans`	Per OCR span text, confidence, bounds, indexes, and raw JSON.
`ax_nodes`	Accessibility nodes with roles, text, bounds, focus, actions, and raw JSON.
`content_units`	Normalized product-facing units derived from AX and OCR.
`ui_events`	Lightweight native event stream.
`capture_triggers`	Coalesced trigger records linking UI events to attempted captures.
`event_transitions`	Classified state changes between pre-frame and post-frame evidence.
`window_snapshots`	macOS window graph snapshots.
`windows`	Individual windows observed in a window graph.
`frame_diffs`	Simplified frame-to-frame changes.
`app_contexts`	Product-object adapters for apps, tabs, docs, terminals, conversations, and other surfaces.
`clipboard_events`	Clipboard metadata without full clipboard text.
`typing_bursts`	Keyboard activity summaries without raw typed characters.
`presence_samples`	Activity samples.
`exclusion_rules`	Privacy and exclusion rules.
`sensitive_regions`	Sensitive visual/text regions and actions taken.
`frame_quality_warnings`	Frame-level evidence warnings.
`frame_text_resolutions`	Active/background text split, quality label, attribution flags, hashes, and the resolution payload used by Continue.
`ai_export_audit`	Safe AI export audit rows.
`local_memory_maintenance`	Runtime counters and maintenance values for storage, capture, cleanup, enrichment, audit, and Continue diagnostics.

Continue tables created by ensure_continue_schema:

Table	Purpose
`continue_schema_migrations`	Continue schema version marker.
`continue_artifacts`	Stable work objects such as browser tabs, conversations, code editors, terminals, PDFs, messages, and docs.
`continue_artifact_observations`	Per-frame or event-derived observations of artifacts.
`continue_task_actions`	Derived local actions such as editing, searching, encountering an error, branching away, or returning to origin, including P6 task-turn scope and semantic-delta fields.
`continue_task_action_events`	Join table from task actions to native UI events.
`continue_ordered_evidence_spans`	P6.02 reading-order evidence spans with AX/OCR/content provenance, region and conversational roles, ownership, geometry, confidence, and rejection reasons.
`continue_salient_turn_evidence`	P6.02 selected and rejected user, agent, and prior-boundary evidence for each observed frame.
`continue_task_turns`	P6.03 current and historical task-turn identity, summary, revision, lifecycle axes, prior relation, and material fingerprint.
`continue_task_turn_evidence`	Field-level task-turn claims linked back to ordered evidence spans and source frames.
`continue_task_turn_relations`	Prior, parent, supersession, and other evidence-backed relationships among task turns.
`continue_task_turn_lifecycle`	Revision-by-revision changes to execution state, current actor, and waiting-on state.
`continue_semantic_moments`	Debounced meaningful changes and event/frame boundaries that can invalidate stale decisions.
`continue_boundary_revisions`	Persisted semantic-boundary revisions used by cache freshness.
`continue_episodes`	Adjacent task actions grouped into local episodes.
`continue_episode_actions`	Episode-to-action join rows.
`continue_episode_artifacts`	Artifact roles inside each episode.
`continue_workstreams`	Durable clusters of related episodes and artifacts.
`continue_workstream_episodes`	Workstream-to-episode join rows.
`continue_workstream_artifacts`	Durable artifact roles inside a workstream.
`continue_workstream_state_snapshots`	Historical workstream state/unresolved-state snapshots.
`continue_workstream_edges`	Relationships among workstreams.
`continue_task_turn_workstream_memberships`	P6.05 task-turn/workstream membership, relation, confidence, selected state, and consistency evidence.
`continue_branch_contexts`	Origin, branch, return, and explicit branch-promotion state for support surfaces.
`continue_open_loops`	Evidence-backed unfinished, blocked, completed, or unclear work between workstreams and candidates.
`continue_open_loop_lifecycle_events`	P6.05 task-turn-scoped open-loop creation, state-transition, reparenting, and closure history.
`continue_open_loop_artifacts`	Artifact roles inside an open loop.
`continue_open_loop_evidence`	Evidence handles supporting an open loop.
`continue_app_activity_segments`	Bounded app/surface activity segments used by scoring.
`continue_activity_classifications`	Local activity classifications and provenance.
`continue_candidates`	Scored continuation candidates.
`continue_decisions`	Persisted Continue decisions and provenance.
`continue_decision_open_events`	Privacy-preserving open attempts/outcomes keyed by decision and source; no raw URL/path payload.
`continue_feedback_events`	Inferred and explicit feedback about a decision.
`continue_feedback_policy_evaluations`	P6.04 provenance, scope, decay, polarity, and allowed-effect decisions for each feedback event and context.
`continue_breadcrumbs`	Manual local next-step notes attached to workstreams.
`continue_evidence_probes`	Bounded requests/attempts to acquire missing evidence.
`continue_memory_cells` / `continue_memory_edges`	Durable support/contradiction memory used during retrieval and ranking.
`continue_pairwise_preferences` / `continue_ranking_priors`	Learned local preference and ranking features derived from feedback/evidence.
`continue_surface_enrichment_attempts`	Normalized bounded weak-surface enrichment attempts.
`continue_surface_snapshots`	Normalized weak-surface state, identity, quality, openability, and missing-evidence snapshots.
`continue_semantic_eligibility_evaluations`	P6.05 current-task relation, freshness, provenance, and primary/support/target eligibility decisions.
`continue_semantic_consistency_evaluations`	P6.05 persisted agreement or conflict across task turn, workstream, open loop, branch, recap, and target layers.
`continue_eval_fixtures`	Local deterministic eval fixture storage.

The Continue schema name is:

smalltalk.continue_memory.v1

Capture Runtime

The active capture runtime is in src-tauri/src/capture.rs.

Main Tauri commands:

Command	Purpose
`start_capture`	Start local memory, create or resume a capture session, launch runtime worker, start native event source, capture initial evidence.
`stop_capture`	Pause local memory, stop worker, finalize session, build stop-time resume-query bundle.
`capture_once`	Manual heavy evidence capture.
`capture_status`	Return local memory status, counts, paths, latest frame, tool availability, and runtime diagnostics.
`delete_all_frames`	Stop runtime and clear the live capture store for a clean slate.
`get_local_memory_diagnostics`	Return storage, row counts, budgets, cleanup potential, and runtime diet counters.
`cleanup_local_memory`	Preview or apply retention cleanup of low-value local evidence.
`dev_reset_local_memory`	Stop runtime, clear live store, optionally clear generated debug exports.
`search_captures`	Search frames with SQLite FTS or return latest frames.
`get_frame`	Load one frame row.
`get_frame_image_variant`	Return screenshot data for preview or full frame rendering.
`get_recent_timeline`	Return recent events, triggers, transitions, and frames.
`get_frame_detail`	Return a frame with AX nodes, OCR spans, content units, transitions, app contexts, sensitive regions, and verification summary.
`validate_frame_consistency`	Verify a frame has expected linked evidence.
`search_content_units`	Search normalized content units.
`add_exclusion_rule`	Add privacy/exclusion rule.
`remove_exclusion_rule`	Remove privacy/exclusion rule.
`list_exclusion_rules`	List privacy/exclusion rules.
`delete_recent_captures`	Delete recent capture rows by range.
`export_debug_episode`	Export debug episode data.
`get_episode_dossier`	Build local episode dossier.
`build_safe_ai_export`	Build redacted safe export for model use.
`get_native_storyboard_dossier`	Legacy local storyboard dossier.
`classify_episode_transitions`	Legacy transition classifier path.
`get_native_resume_card`	Legacy local resume-card path.
`build_resume_query_bundle`	Build bounded resume-query bundle.
`build_session_index`	Build stop/session index metadata.
`run_cloud_resume`	Stop/resume-query OpenAI path. Separate from Continue.
`get_cloud_resume_status`	Report OpenAI key presence and model for cloud resume.
`get_continue_decision`	Core product decision; optionally schedules an explicit lean audit.
`get_continue_decision_trace`	Inspect the persisted decision pipeline without dumping broad unsafe history.
`get_continue_memory_status`	Return semantic-memory counts and layer state.
`assess_continue_evidence_sufficiency` / `request_more_continue_evidence`	Evaluate missing evidence and perform bounded probe requests.
`record_continue_feedback` / `infer_continue_feedback`	Persist explicit feedback or infer it from later local activity/open lifecycle.
`run_continue_eval` / `run_continue_replay_eval`	Deterministic fixture and replay evaluation.
`get_island_continue_state` / `perform_island_continue_action`	Typed island state gateway and source-aware action dispatch.
`open_resume_point`	Strict product open by persisted decision id plus explicitly gated diagnostic compatibility opens.

start_native_capture, stop_native_capture, capture_once_v2, and get_frame_v2 are compatibility wrappers over the active capture implementation.

Capture Status Contract

capture_status returns:

running
frame_count
recent_app_labels
signal_count
event_count
transition_count
content_unit_count
session_count
active_session
latest_session
last_export
started_at
last_error
latest_frame
skipped_samples
last_skipped_at
data_dir
database_path
screenshot_tool
accessibility_tool
ocr_tool
runtime_diagnostics

runtime_diagnostics contains:

heavy_captures_stored
heavy_captures_skipped
heavy_captures_skipped_budget
heavy_captures_skipped_dedupe
heavy_captures_skipped_privacy
heavy_captures_skipped_cancellation
heavy_captures_skipped_smalltalk_self
events_aggregated
ocr_runs
ax_snapshots
continue_normal_calls
continue_rebuild_calls
decision_cache_hits
weak-surface enrichment attempt/success/skip/failure counters
latest weak-surface attempt and snapshot ids when present

These fields matter because the current product should not measure memory only by screenshot frames. It also has lightweight local signals and Continue objects.

Local Memory Diagnostics Contract

get_local_memory_diagnostics returns:

database_path
captured_root
database_bytes
snapshot_bytes
safe_export_bytes
frame_count
event_count
heavy_evidence_rows
continue_object_counts
low_value_duplicate_frames
excess_low_value_events
self_capture_frames
self_capture_events
decision_linked_frames
estimated_cleanup_potential_bytes
oldest_retained_frame_ms
latest_frame_ms
cleanup_last_run_ms
cleanup_last_result
budgets
runtime_diagnostics

heavy_evidence_rows includes:

content_units
ax_nodes
ocr_text_rows
ocr_spans
app_contexts
window_snapshots
windows

continue_object_counts includes:

artifacts
artifact_observations
semantic_moments
task_actions
episodes
workstreams
open_loops
candidates
decisions
open_events
feedback_events
breadcrumbs

budgets includes:

min_important_capture_interval_ms
min_low_value_capture_interval_ms
idle_capture_interval_ms
rolling_window_ms
max_screenshots_per_10_minutes
max_screenshots_per_surface_without_change
max_snapshot_dir_bytes
max_retained_low_value_duplicate_frames
max_retained_low_value_ui_events
max_diagnostic_rows_per_cleanup

Current budget constants in capture.rs:

Constant	Value	Meaning
`MIN_IMPORTANT_CAPTURE_INTERVAL`	4 seconds	Minimum spacing for important heavy captures.
`MIN_LOW_VALUE_CAPTURE_INTERVAL`	45 seconds	Minimum spacing for low-value heavy captures.
`IDLE_CAPTURE_INTERVAL`	120 seconds	Idle capture interval.
`CAPTURE_BUDGET_ROLLING_WINDOW_MS`	10 minutes	Rolling budget window.
`MAX_SCREENSHOT_FRAMES_PER_10_MINUTES`	24	Heavy screenshot budget per rolling window.
`MAX_SCREENSHOT_FRAMES_PER_SURFACE_WITHOUT_CHANGE`	3	Maximum retained heavy frames for unchanged surface.
`MAX_LOCAL_SNAPSHOT_DIR_BYTES`	512 MB	Target local snapshot directory budget.
`MAX_RETAINED_LOW_VALUE_DUPLICATE_FRAMES`	400	Duplicate cleanup threshold.
`MAX_RETAINED_LOW_VALUE_UI_EVENTS`	5,000	Retain the newest low-value scroll/AX event rows before older rows become cleanup candidates.
`MAX_STORED_DIAGNOSTIC_ROWS_PER_CLEANUP`	5,000	Cleanup diagnostic cap.
`MAX_RETAINED_FRAME_AGE_MS`	7 days	Retention age for frame cleanup candidates.
`MAX_RETAINED_CONTINUE_DECISION_AGE_MS`	24 hours	Retention age used to protect recent Continue decisions.
`LOCAL_SIGNAL_BUCKET_WINDOW_MS`	30 seconds	Window for aggregating recent local signals.
`LOCAL_SIGNAL_RECENT_WINDOW_MS`	20 minutes	Recent event window used for visible local signal counts.
`MAX_LOCAL_SIGNAL_EVENTS`	600	Maximum raw local signal events considered.
`MAX_VISIBLE_SIGNAL_MOMENTS`	48	Product-facing cap for bucketed signal moments.
`ISLAND_EVENT_STATUS_INTERVAL`	900 ms	Minimum interval for refreshing island/status after native events.

These are pressure controls, not a promise that every hour stores exactly the same amount. Important triggers such as app/window changes, clipboard activity, explicit captures, session start, and error-bearing surfaces can exceed the low-value screenshot cadence. Conversely, unchanged surfaces can produce many lightweight events and very few heavy frames.

For an hour of sustained low-value activity, the 45-second spacing permits at most about 80 heavy-capture attempts before same-surface dedupe and storage pressure reduce it further. The 24-frame rolling count can be more restrictive when important frames also occurred in the preceding 10 minutes, because those frames count toward the observed total even though important/manual/error captures themselves bypass the low-value rejection. This is not a global frame maximum. The event stream is also not capped at 5,000 during capture; 5,000 is the retention threshold used by cleanup for low-value scroll/AX rows.

What Happens During An Hour-Long Session

There is no special one-hour mode and no hour-boundary rollover. A long session follows the same event-driven lifecycle continuously:

start_capture creates a capture_sessions row, starts the Rust worker and native Swift event source, and attempts a session_start heavy frame.
Native events append lightweight ui_events, typing-burst, clipboard, trigger, and transition evidence linked to the active session.
Events are throttled/coalesced. They can update freshness and semantic memory without a screenshot.
When a trigger settles, the worker decides whether heavy evidence is valuable. Important/manual/error triggers are favored; low-value/unchanged/self/private activity is skipped or deduped.
Accepted heavy evidence writes day-partitioned JPEG assets and normalized SQLite rows. AX is preferred; OCR runs only when AX is missing/thin; active/background attribution is persisted.
Weak surfaces can create bounded enrichment attempts/snapshots from already persisted local evidence.
Continue can run at any time without stopping the session. It incrementally rebuilds semantic rows, reuses a cache only when the full watermark is unchanged, and persists the exact decision.
Status/diagnostic polling uses short read-only connections where possible. Writes use WAL plus a busy timeout; decision work is serialized so the island and main card do not run conflicting rebuilds.
No continue_outputs/ folder is produced merely because capture is running. Only an explicit audit-enabled Continue schedules a compact asynchronous bundle.
stop_capture stops the event source/worker, finalizes the session row/counts, and can build the older bounded stop-time resume_query_exports diagnostic bundle. Stop is not required for Continue.
Historical data remains in the shared local database after stop. Cleanup is user-triggered and evidence-aware; it is not tied to session duration.

The main growth sources are JPEG assets and heavy AX/OCR/window/content rows, not the small event metadata alone. The product exposes database bytes, snapshot bytes, row counts, skip counters, excess low-value rows, cleanup potential, and oldest/latest retained evidence so an hour-long session can be evaluated from measured local state rather than estimated from duration.

SQLite Concurrency And Durability

All database access goes through shared helpers:

writable opens configure journal_mode = WAL and synchronous = NORMAL;
writable and read-only opens set a 30-second SQLite busy timeout;
polling/watermark paths use SQLITE_OPEN_READ_ONLY where they do not need schema mutation;
CONTINUE_DECISION_LOCK permits one in-process get_continue_decision at a time;
cleanup checkpoints the WAL and can optionally run VACUUM;
background audit export reuses the configured connection helpers instead of ad hoc SQLite opens.

These changes address the observed Continue failed: database is locked class of failure during active sessions. They reduce lock contention but do not turn SQLite into a multi-process server: another process holding an exclusive write or an unexpected crash can still produce an error, which must remain visible rather than being silently treated as a valid Continue result.

The July 11 legacy-schema repair fixes a separate startup failure. ensure_continue_schema now adds the nullable continue_task_actions.task_turn_id column before it creates idx_continue_task_actions_task_turn. Existing pre-P6 databases can therefore migrate in order instead of failing with no such column: task_turn_id. The React error productizer also treats missing-column, missing-table, malformed-schema, and database-schema failures as a local database update problem. The earlier on-screen Accessibility warning was a UI classification error caused by schema SQL that happened to contain Accessibility-related text; it was not evidence that macOS Accessibility permission had failed.

Capture Trigger Model

Smalltalk uses lightweight native events to decide when heavy evidence is worth storing.

On macOS, src-tauri/scripts/capture_events.swift is compiled into the app-data helpers/ directory and runs as a child process. It combines:

NSWorkspace.didActivateApplicationNotification for app changes.
AXObserver notifications for focused-window, focused-element, value, selected-text, window, and title changes.
A CGEvent tap for mouse-down, categorized key-down, and scroll-wheel activity.
Pasteboard change polling for clipboard metadata.

The helper caches frontmost app/window lookup for 250 ms, coalesces scroll emission to roughly one row per 650 ms, and applies notification-specific AX throttles. Rust applies a second storage throttle: scroll rows are retained no faster than 650 ms, generic AX rows no faster than 900 ms, and AX value-change rows no faster than 1.8 seconds for the same surface. This keeps long sessions from turning native notification churn into an unbounded database write stream.

Heavy captures are expensive because they can include screenshots, AX snapshots, window graphs, OCR, content-unit extraction, and SQLite writes. The product should therefore treat heavy frames as bounded evidence, not as the only memory.

Trigger types include:

Trigger	Source	Meaning
`session_start`	Rust runtime	Initial frame when local memory starts.
`manual`	UI command	User explicitly requested evidence capture.
`app_switch`	Native event helper	Frontmost app changed.
`window_focus`	Native event helper	Focused window changed.
`accessibility_change`	Native event helper	AX notification indicated UI content changed.
`click`	Native event helper	Pointer click.
`typing_pause`	Native event helper	Keyboard activity paused.
`scroll_stop`	Native event helper	Scrolling settled.
`clipboard`	Native event helper	Clipboard metadata changed.
`idle`	Runtime timer	Idle fallback when useful capture has not happened recently.
`event_burst`	Runtime coalescing	Multiple event kinds merged before capture fired.

The runtime stores event rows even when a heavy screenshot is skipped. Continue can use event-backed evidence and synthetic ids such as event-<hash> to avoid collapsing local memory to only screenshot frames.

Visible local signal counts are intentionally bounded. The status path looks at recent event rows, buckets them into signal moments, and caps the product-facing count so a long event stream does not make the app look heavier or more precise than it is.

Native event ingestion can refresh capture-status and the floating island even when no heavy frame is stored. This keeps local-memory state alive during event-only periods without requiring continuous screenshots.

Trigger coalescing keeps one pending bucket. Key activity settles into typing_pause after about 850 ms; scrolling settles into scroll_stop after about 500 ms. More events arriving before the deadline extend the bucket, and mixed trigger types become event_burst. The event rows remain evidence even if the eventual heavy capture is skipped.

Native Capture Pipeline

A stored heavy frame follows this pipeline:

Resolve app-data paths, ensure the database/schema, and create the day-partitioned snapshot directory.
Reuse pre-collected Accessibility context when the trigger already has it, otherwise collect foreground app, window, URL/path, selected/focused state, nodes, and text.
Compute a semantic fingerprint from the foreground context.
Apply privacy and exclusion rules before image capture; return a privacy skip without writing a frame when capture is disallowed.
Apply the pre-capture pressure gate. Manual/session-start and visible-error captures bypass the low-value gate; Smalltalk self-observation, exhausted rolling/snapshot budgets, or too many unchanged frames can skip heavy capture.
Collect the macOS window graph when available.
Capture a full-display JPEG with ScreenCaptureKit first and /usr/sbin/screencapture as fallback.
Attempt a ScreenCaptureKit active-window JPEG when a window id is known.
Hash the screenshot and record dimensions/provider/scope/SCK provenance.
Determine whether Accessibility text is strong or thin.
Run OCR against the active-window crop when present, otherwise the full display, only when Accessibility is missing or thin.
Attribute OCR spans to windows/surfaces, separate active owned text from background/display-only text, and persist the resolution quality/flags.
Resolve text_source, full_text, active content_hash, and image hash.
Apply final image-plus-content deduplication. A frame is skipped only when both image and content match; skipped temporary JPEGs are deleted.
Insert the frame and FTS row in SQLite.
Persist OCR text/spans, AX nodes, window graph, app contexts, content units, sensitive regions, presence, frame text resolution, quality warnings, frame diff, capture-trigger result, and transition rows.
Run bounded weak-surface enrichment over the persisted frame and store an attempt/snapshot when applicable.
Increment runtime counters and emit capture-frame to React.

Manual captures bypass normal duplicate suppression because the user explicitly requested evidence.

Event and idle captures can be skipped for budget, duplicate, privacy, cancellation, or Smalltalk self-observation reasons. These skips increment runtime counters.

Important trigger types are manual, session_start, app_switch, window_focus, and clipboard. They are not rejected by the 24-per-10-minute low-value budget. Non-important captures are skipped when the rolling frame count is already 24, the snapshot directory is over 512 MB, or an unchanged semantic surface already has three recent frames. Any surface with a locally detected error signal bypasses those low-value pressure gates so blocker evidence is not discarded merely to save space.

Screenshot Capture

The current code has ScreenCaptureKit metadata fields on frames:

sck_display_id
sck_window_id
sck_owning_bundle_id
sck_filter_summary_json
sck_configuration_summary_json
sck_frame_metadata_json
sck_capture_mode
sck_audio_policy

The UI displays capture provider and SCK scope in diagnostics.

ScreenCaptureKit is the preferred live provider, implemented by src-tauri/scripts/sck_screenshot.swift through SCScreenshotManager.captureImage. The provider is still-image capture inside the event-driven scheduler, not continuous screen recording. A successful primary-path frame reports capture_provider = screen_capture_kit; an active-window capture uses scope = active_window and normally has a _window.jpg asset. If the helper fails, Smalltalk can fall back to the command-line screenshot path and records that provider instead of pretending SCK was used.

The older command-line screenshot fallback is:

/usr/sbin/screencapture -x -t jpg

The product should treat screenshot provider details as diagnostics. The product answer should talk about workstreams and evidence quality, not which screenshot backend ran.

Each CaptureFrame includes:

id
captured_at
snapshot_path
app_name
window_name
browser_url
document_path
focused
capture_trigger
text_source
accessibility_text
accessibility_tree_json
full_text
content_hash
image_hash
capture_provider
scope
display_id
window_id
app_pid
app_bundle_id
screen_scale
pixel_width
pixel_height
full_screenshot_path
active_window_crop_path
active_element_crop_path
phash
privacy_status
capture_trigger_id
previous_frame_id
session_id
ScreenCaptureKit metadata fields

Text Extraction

Smalltalk extracts text through Accessibility first and OCR second.

Accessibility

The primary helper is:

src-tauri/scripts/accessibility_snapshot.swift

The Rust backend compiles or prepares Swift helpers through helper setup code and parses structured helper output.

Accessibility captures:

frontmost app name
app PID
app bundle id
focused window title
window id
browser URL when available
document path from AXDocument when available
selected text
focused node
Accessibility nodes
node roles
node labels
node values
descriptions
bounds
actions
depth

Accessibility nodes in Rust include:

local_id
parent_id
role
subrole
role_description
title
value
description
help
identifier
document
url
selected_text
selected_text_range
visible_character_range
number_of_characters
focused
enabled
selected
bounds
actions
children_count
text
depth

If the Swift helper fails or returns weak signal, the backend has an AppleScript fallback path embedded in capture.rs.

Thin Accessibility

Accessibility is treated as thin when the visible surface is likely canvas-heavy, browser-chrome-heavy, or low-text. Thin Accessibility is not discarded. It causes OCR to run and can produce a hybrid text source.

Examples of surfaces where Accessibility may be thin:

Google Docs
Google Sheets
Google Slides
Figma
Excalidraw
Miro
Canva
tldraw
other canvas-heavy or custom-rendered UIs

Thinness can also come from too little content-like text or too much toolbar/chrome-like text.

OCR

OCR runs when Accessibility text is missing or thin.

The primary OCR helper is:

src-tauri/scripts/vision_ocr.swift

Apple Vision OCR is preferred on macOS. tesseract is the fallback when available.

OCR rows are stored in:

ocr_text
ocr_spans

OCR spans preserve text, confidence, block/line/word indexes, pixel bounds, normalized bounds, and raw JSON.

Text Source Resolution

The backend preserves the legacy text_source/full_text view for search and inspection, but Continue uses the newer frame_text_resolutions attribution when available. That row contains active text, background text, diagnostic text, full_text_quality, quality flags, resolution JSON, and active-content hash.

Evidence state	`text_source`	`full_text`
Accessibility strong	`accessibility`	Accessibility text.
Accessibility thin plus OCR	`hybrid`	Accessibility text plus OCR text.
OCR only	`ocr`	OCR text.
Neither source available	`null`	`null`.

full_text powers FTS search, frame inspection, content-unit extraction, and downstream Continue evidence.

For Continue, active_frame_text_for_continue prefers attributed active text. Raw full_text is blocked as primary task evidence when quality is mixed_active_and_background, background_only, display_only_unattributed, or unknown. Background OCR can therefore remain locally inspectable without being allowed to invent the current task, error, or return target.

Content Units

Content units are normalized evidence units derived from Accessibility nodes and OCR spans.

They are stored in content_units with:

source: ax or ocr
unit_type: button, input, link, menu item, table cell, image, heading, paragraph, or unknown
text and text hash
semantic role
linked AX node id or OCR span ids
bounds
confidence
raw JSON

Semantic roles include:

toolbar
browser_chrome
app_sidebar
composer
error
search_result
code_editor
terminal_output
chat_message
main_content

Focused nodes get higher confidence than generic nodes. OCR-only units are lower confidence than Accessibility-derived units unless Accessibility is absent.

Window And App Context

The window graph helper is:

src-tauri/scripts/window_snapshot.swift

Window snapshots store:

active window id
active app PID
active app bundle id
screen count
window rows

Individual windows store:

CoreGraphics window id
owner PID
owner name
bundle id
window title
layer
alpha
onscreen flag
active flag
bounds
workspace
raw metadata

App contexts adapt raw app/window/URL/path evidence into product objects.

Surface	Object type
ChatGPT, Claude, or similar browser conversation	`chat_conversation`
Normal browser tab	`browser_tab`
Cursor, VS Code, Xcode, IntelliJ	`code_editor`
Terminal, iTerm, Warp	`terminal`
Preview or PDF surface	`pdf`
Finder	`finder`
Slack, Discord, Messages, WhatsApp	`messaging`
Notion, Linear, Notes	`notes_doc`
Unknown app	`unknown`

App contexts are evidence objects. Continue can use them to resolve artifacts, but the app context itself is not automatically a return target.

Events, Typing, Clipboard, Transitions, And Diffs

ui_events store native event metadata:

event id
timestamp
event type
app name
window title
key category
pointer location
scroll deltas
modifiers
repeat flags
payload JSON
session id when available

Keyboard events do not store raw typed characters. They store categories such as:

char
enter
backspace
shortcut
modifier
escape
arrow

typing_bursts summarize keyboard activity without raw typed text. They can record counts, paste count, enter count, commit signal, and whether a burst looked committed.

clipboard_events store clipboard metadata without full clipboard text.

capture_triggers connect events to heavy capture attempts. They store:

trigger id
trigger type
event ids that caused the trigger
settle delay
dedupe policy
pre-frame id
post-frame id
status
errors

event_transitions summarize what happened between pre-frame and post-frame evidence.

Transition labels include:

switched_app
scrolled_to_new_section
entered_input
copying_evidence
same_screen_idle
continuing_same_task
new_task
unknown

Higher-level story labels used by local resume/storyboard paths include:

returning_to_previous_task
verification_branch
possible_distraction
background_media

frame_diffs store simplified changes such as same app/window, changed text hashes, diff type, confidence, and summary.

Search

The UI calls:

await invoke("search_captures", {
  query,
  limit,
  sessionId
});

If the query is empty, the backend returns latest frames for the active or latest session. If the query has terms, the backend builds an FTS query across:

full_text
app_name
window_name
browser_url
document_path

Results include:

frame row
SQLite FTS snippet
BM25 rank

Search is a diagnostic and evidence-inspection tool. It should not be the primary product model.

Continue Architecture

Native Continue is implemented in src-tauri/src/continuation.rs and exposed through command wrappers in src-tauri/src/capture.rs.

Continue is additive on top of the local capture store. It does not require Stop Session. It does not use the browser extension. It does not send broad raw history to a model.

The implementation is easiest to understand as eight stages:

Evidence substrate.
Text attribution and weak-surface enrichment.
Semantic memory: artifacts, observations, actions, semantic moments, and boundary revisions.
Episodes, workstreams, branch contexts, state snapshots, and open loops.
Current-surface resolution, freshness, candidate generation, retrieval, scoring, feedback, and quality gates.
Optional bounded micro-inference over locally supplied candidates, with local validation and fallback.
P5 activity recap over bounded decision/evidence facts: timeline stitching, grounded work and location labels, detours, state left behind, next action, optional validated phrasing, and narrative memory.
Persisted decision consumed by React, the native island, strict open, feedback, trace, eval, and optional audit output.

Layer 1: Evidence Substrate

The substrate is local SQLite evidence from capture:

frames
app contexts
content units
UI events
triggers
transitions
frame diffs
typing bursts
clipboard metadata
privacy markers
search indexes
window graph rows
OCR rows
AX rows
frame text resolutions
weak-surface enrichment attempts and snapshots

This layer is factual. It should not infer broad intent.

Layer 2: Semantic Memory

The semantic memory layer is rebuilt with:

rebuild_continue_second_layer

It performs:

Load evidence frames for a session/lookback/limit.
Clear second-layer rows for those frames.
Load normalized weak-surface snapshots and event-only surface evidence in addition to frames.
Resolve a stable artifact for each frame, snapshot, or event-backed evidence item.
Upsert continue_artifacts and link snapshots back to resolved artifacts.
Upsert continue_artifact_observations.
Extract task actions and classify conservative branch roles.
Collapse repeated task actions.
Insert continue_task_actions and task-action event links.
Build semantic moments and boundary revisions from meaningful frame/event deltas.

Result fields include:

processed_frames
artifact_count
observation_count
task_action_count
start_frame_id
end_frame_id

Artifact Resolution

Artifacts are stable local work objects.

Artifact kinds:

browser_tab
chat_conversation
code_editor
terminal
pdf
finder
messaging
notes_doc
unknown

Artifact identity prefers durable keys. The priority is:

Meaningful safe browser URL.
Document path.
App-context object id.
App plus window title.
Stable hash fallback.

Weak editor/terminal/native-agent surfaces use privacy-safe identity adapters when a direct URL/path is not available. For example, code-editor identity can combine a hashed repository root with a hashed relative file identity. Missing identity fields and merge keys are persisted for auditability; raw selected text is hashed rather than stored by the enrichment layer.

Artifacts store:

id
artifact_kind
stable_key
app_name
bundle_id
window_title
browser_url
document_path
display_title
first/last seen frame ids
first/last seen timestamps
identity_confidence
evidence_quality
privacy_status
openability
timestamps

Evidence quality values:

strong
medium
thin
unknown

Openability values:

openable
frame_fallback
blocked
unknown

Text source values:

accessibility
ocr
hybrid
missing

Task Actions

Task actions are derived from local evidence. They do not store raw typed characters.

Action kinds:

reading
editing
composing
searching
copying_evidence
reviewing_output
running_command
observing_command_output
encountering_error
navigating
switching_context
branching_away
returning_to_origin
idle_after_progress
messaging_interrupt
verification_branch
possible_distraction
unknown

Action roles:

primary
support
branch
return
interrupt
unknown

Task actions store:

action id
frame id
previous frame id
artifact id
secondary artifact id
action kind
action role
trigger type
transition label
evidence event ids
confidence
local reason
created timestamp
collapse count
first frame id
last frame id
strongest frame id

The classifier is intentionally conservative. Search branches, verification tabs, and messaging surfaces should usually become support evidence, not the default return target.

Each branch action can carry a deterministic taxonomy and promotion state. Origin -> branch -> return relationships are persisted in continue_branch_contexts. Openability, a good title, recency, or the model's preference are not promotion evidence. Public promotion requires newer local proof such as direct editing/composing, a visible unresolved blocker, sustained work after abandoning the origin, explicit correction/acceptance, a fresh breadcrumb, or a confident primary action on the branch.

Weak-Surface Enrichment

Weak surfaces are screens where a normal URL/path/title is missing or insufficient, especially code editors, terminals, Codex/native agent windows, and custom-rendered tools. The enrichment subsystem lives in src-tauri/src/continuation/enrichment.rs.

It is bounded and metadata-driven; it is not another capture loop. Focus/event triggers can schedule an attempt, and get_continue_decision can perform a bounded synchronous fallback before deciding. Adapters read already persisted frames, AX/OCR/content units, app contexts, recent events, typing summaries, clipboard metadata, and window state.

Normalized rows are stored in:

continue_surface_enrichment_attempts: reason, domain/adapter, budget/privacy/outcome, timestamps, missing evidence, and optional snapshot link.
continue_surface_snapshots: stable surface key, app/window identity, hashed repo/file/conversation hints, task/activity state, bounded visible sample, quality, identity confidence, openability, privacy, missing evidence, and artifact link.

evaluate_surface_snapshot_quality deterministically produces evidence quality, identity confidence, candidate eligibility, stale-target suppression strength, openability, missing-evidence labels, and warnings. Thin/unknown snapshots may describe fresh current work but cannot become primary return targets merely because they are recent.

Semantic Moments And Boundary Revisions

Raw event/frame sequences are converted into meaningful moments such as content change, progress, event-only activity, task/surface transition, and invalidating evidence. continue_semantic_moments stores the evidence-backed delta; continue_boundary_revisions stores revisions that invalidate stale decisions. This lets a meaningful event change Continue freshness even when the screenshot count did not change.

Event-Only Continue Evidence

The current code includes regression coverage for Continue using event-only moments without new screenshot frames. This matters because the product no longer wants to depend on a constant stream of screenshots.

Event-only evidence can:

update latest evidence timestamp
influence cached decision freshness
create artifact observations or actions using synthetic event ids
help recent app labels and moment counts avoid getting stuck
preserve privacy and storage budgets by not forcing heavy screenshots for every interaction

Continue must treat frames and events as evidence, but only heavy frames have screenshot previews.

Layer 3: Episodes And Workstreams

The workstream layer is rebuilt with:

rebuild_continue_third_layer

It performs:

Load task actions.
Clear third-layer rows.
Group actions into episodes.
Assign artifact roles inside episodes.
Cluster episodes into workstreams.
Assign durable artifact roles inside workstreams.
Store unresolved state and historical workstream state snapshots.
Build workstream relationships and branch origin/return context.
Build evidence-backed open loops between workstreams and candidates.
Insert episode/workstream/artifact join rows.

Result fields include:

processed_actions
episode_count
episode_action_count
episode_artifact_count
workstream_count
workstream_episode_count
workstream_artifact_count
start_frame_id
end_frame_id

Episode states:

open
closed
merged
discarded

Episode artifact roles:

primary_target
source_evidence
branch_support
output_verification
blocker
interruption
current_focus_only
unknown

Workstream states:

active
suspended
resumed
background
stale
abandoned

Workstream sources:

local_heuristic
micro_inference

A workstream stores:

workstream id
state
title candidate
inferred intent
primary artifact id
created timestamp
last active timestamp
suspended timestamp
confidence
unresolved signal
source

Unresolved signals are local JSON or local string reasons. They should not become raw product copy. The frontend productizes common internal labels before display.

Examples of unresolved states:

idle after meaningful progress
visible error still unresolved
draft or composer active
verification branch without return
search branch without return
copied evidence not yet applied

Open loops summarize the actual continuation boundary: last concrete progress, unfinished/blocked/completed state, next evidence-backed action when known, current-focus relation, artifact roles, quality, and supporting evidence. Candidate generation uses open loops rather than treating every recent surface as equally resumable.

Layer 4: Local Continue Decision

The main command is:

get_continue_decision

Default backend request values:

lookback_ms: 45 minutes
limit: 700
mode: normal
rebuild_layers: false
micro_inference_enabled: true
activity_recap_model_enabled: false
max_candidates_for_model: 5
audit_output_enabled: false

The frontend normal path sends:

await invoke("get_continue_decision", {
  input: {
    mode: "normal",
    rebuild_layers: false,
    micro_inference_enabled: true,
    activity_recap_model_enabled: trigger === "manual",
    max_candidates_for_model: 5,
    audit_output_enabled: options.writeAudit === true
  }
});

The developer diagnostic rebuild path sends:

await invoke("get_continue_decision", {
  input: {
    mode: "rebuild",
    rebuild_layers: true,
    micro_inference_enabled: true,
    activity_recap_model_enabled: true,
    max_candidates_for_model: 5,
    audit_output_enabled: true
  }
});

effective_continue_decision_mode treats rebuild_layers: true as rebuild. Modes rebuild, force_rebuild, and diagnostic_rebuild force rebuild. Other modes are normal.

Normal mode can reuse a cached decision when no newer local evidence exists. Cache hits increment decision_cache_hits. Normal calls increment continue_normal_calls. Rebuild calls increment continue_rebuild_calls.

The Tauri wrapper serializes get_continue_decision with CONTINUE_DECISION_LOCK. This avoids two overlapping main-card/island decision rebuilds fighting over the same SQLite writer. Watermark/status reads use read-only connections, and React's post-decision island sync passes allow_refresh: false so it does not immediately launch a second decision.

get_continue_decision does this:

Ensure Continue schema.
Normalize request defaults.
Determine normal versus rebuild mode.
Build an evidence watermark from frames, events, semantic moments, boundary revisions, feedback, opens, and surface snapshots.
Try a cached decision only when inference policy, evidence and recap watermarks, recap policy fingerprint, boundary state, feedback/open watermarks, and freshness still match.
Infer matured pending feedback for prior open events when no cache is reused.
Run bounded pre-decision weak-surface enrichment when current evidence needs it.
Rebuild semantic layer 2 and workstream layer 3 incrementally when needed.
Resolve current surface by fusing frames, events, artifact observations, app contexts, window state, typing activity, and enriched snapshots.
Derive active_current_work_unresolved separately from any return target.
Load workstreams, state snapshots, graph relationships, and open loops.
Generate candidates, including fresh non-openable continue_current_work candidates.
Apply app-activity features and retrieve local memory support/contradiction cells.
Apply ranking priors, feedback aggregation, hard suppression, branch-promotion eligibility, scoring, and risk caps.
Persist locally generated candidates and select the best eligible local candidate.
Evaluate the quality gate and initial output mode: strong_continue, thin_continue, or no_clear_continuation.
Build a candidate-bounded model pack after removing feedback-suppressed and unpromoted branch candidates.
Run micro-inference when enabled and eligible candidates exist, then validate the selected ids, semantics, evidence quality, feedback state, branch state, and public copy.
Build the evidence-freshness ledger and suppress stale target revival when fresher current work exists.
Compose locally governed handoff copy and gate public return_target / resume_work_target independently from diagnostic candidates.
Build a stable decision id and persist the target-selection part of continue_decisions when appropriate.
Build the bounded P5 recap inputs, stitch recent activity, infer grounded work/location labels, classify detours/support, and synthesize last/unfinished state and next action.
Optionally run bounded recap phrasing when activity_recap_model_enabled is true, then validate every rewritten claim against supplied terms, evidence handles, confidence, detour identity, and target policy.
Reuse or persist the exact recap, decision proof, recap watermark, and recap policy fingerprint; promote only validated medium/high-confidence narrative memory with stable non-memory anchors.
Return the decision, activity recap, current-work fact, quality signals, anchors, support evidence, alternatives, freshness, retrieval, provenance, warnings, and optional audit path.
If and only if audit_output_enabled is true, schedule the lean proof-first audit asynchronously after the decision is ready.

Candidate kinds:

continue_edit
continue_current_work
return_to_primary_artifact
resolve_error
verify_output
continue_reply
read_next_source
finish_search
rerun_command
resume_chat_reasoning
evidence_only

Scoring components:

actionability_score
primary_target_score
unresolved_score
branch_origin_score
evidence_quality_score
recency_score
openability_score
privacy_safety_score
app-activity, memory support/contradiction, feedback-prior, work-value, resume-likelihood, divergence, objective-relation, interaction-depth, and evidence-sufficiency features

A candidate stores:

candidate id
workstream id
target artifact
candidate kind
last meaningful action
evidence frame id
supporting episode id
total score
score components
local reason
missing evidence
warnings
resume work target

Branch and support targets can be evidence without being default return targets.

Hard suppression is applied before sorting/selection. Candidates rejected by feedback or blocked branch-promotion state are excluded from public alternatives and from the model pack, and a live suppression check runs again in open_resume_point so an old persisted decision cannot reopen a newly rejected target.

Continue Decision Result

ContinueDecisionResult includes:

decision_id
mode
cache_hit
cache_bypass_reasons
source
model
response_id
current_focus
active_current_work_unresolved
p0_quality_signals
current_activity
current_task_turn
selected_workstream
semantic_graph_policy_version
cross_layer_consistency
direct_target_policy
target_truth
evidence_preview
confidence_vector
confidence_summary
legacy_confidence_derivation
return_target
resume_work_target
candidate_kind
last_meaningful_action
unresolved_state
next_action
confidence
confidence_label
evidence_anchors
missing_evidence
warnings
validation_failures
alternatives
generated_candidates
validation_status
handoff
support_evidence
feedback/open watermarks and suppression/filter counters
branch-selection/filter counters and validation failures
continue_output_mode
evidence watermark and latest boundary revision
current_surface_resolution
evidence_freshness_ledger
Continue dossier and memory-retrieval report
observe-before-decide and weak-surface enrichment diagnostics
app-activity summary and quality gate
activity_recap
answer
activity_recap_watermark_hash
micro-inference requested/attempted/result-kind fields
optional continue_output_path

Decision sources:

local_scorer
cloud_micro_inference
local_fallback

Validation statuses:

valid
fallback
rejected
thin_evidence

Confidence labels are derived from numeric confidence. Low confidence should be presented as best available evidence, not as certainty.

P6.06 adds the internal smalltalk.continue_confidence.v2 contract. It keeps surface identity, active-window ownership, region/speaker/turn attribution, latest goal and task object, execution/current-actor/waiting-on state, prior relation, workstream/branch/open-loop alignment, recap support, and direct-target identity/openability/policy as separate evidence-linked dimensions. Claim confidence is the minimum of its critical dimensions, never an average that lets a strong app identity hide a missing task or target fact.

Compatibility fields are explicit derivations: legacy confidence and confidence_label describe the bounded activity-recap claim; activity_confidence derives from that same recap claim; target_confidence derives independently from the direct-target claim. A useful task recap may therefore coexist with target_confidence = none and null public target fields. Observe-before-decide targets a missing critical dimension, records per-probe outcomes, and reruns only after non-stale material evidence changes that dimension. Timeout, privacy block, failure, and successful-no-change outcomes retain the missing evidence and do not emit an evidence-refreshed warning.

P6.07 makes smalltalk.activity_recap_task_truth.v1 the canonical semantic input to both local recap composition and optional model phrasing. The truth pack fixes the current task-turn id/revision, bounded task identity, execution/current-actor/waiting-on axes, prior relation, selected workstream/segment/loop consistency, claim confidence caps, task-turn evidence handles, and the direct-target policy. Historical/support/detour material remains role-labeled context; superseded, unrelated, feedback-rejected, and otherwise ineligible wording is retained only in the local validator as hashes/reason codes and is not sent to the model.

The deterministic local recap is implemented to start from the latest user goal/task object, preserve task and target confidence separately, treat prior completion as history rather than current state, and emit why_no_safe_target instead of target-shaped frame-fallback copy. The optional model must copy the task identity and target policy exactly and provide per-claim evidence handles and confidence at or below local caps. Validation distinguishes copy-only repair from temporal, workstream, ineligible-source, unsupported-claim, and target-policy rejection; every semantic rejection falls back to the local task-truth recap. This is an architectural safety guarantee, not proof that the fallback is useful: when upstream real-surface role resolution chooses the wrong task turn, the local recap faithfully phrases the wrong or empty truth. The July 11 live Codex audit below demonstrates that remaining failure.

Recap proof/cache policy is versioned as P6.07. Proof output includes task truth, eligible/rejected semantic-source diagnostics, the deterministic local recap, claim-to-evidence mapping, model/local identity parity, validation/fallback details, and the available quality-gate result. Promoted recap memory is keyed and tagged with task-turn identity/revision, semantic-consistency provenance, and validator policy; legacy or incompatible recap memory is ignored conservatively and cannot replace a thin current task.

P6.09 adds an explicit release verdict to the accuracy report. A green phase milestone is not a release claim: release_gate.passed also requires the frozen broad-corpus, human-review, partition/holdout, zero-tolerance, calibration, performance, privacy, and manual macOS gates. The current seven synthetic Capture-button cases pass their deterministic semantic checks, but P6 remains release-incomplete because the repository lacks the required 100 independently human-reviewed cases and locked holdout, several required metrics have no positive labeled samples, calibration is undersized, the latest debug replay exceeded the frozen p95 regression budget, and native interruption-recovery QA is not complete. The authoritative current audit is docs/phases/p6-task-turn-accuracy/p6-09-completion-audit.md; the machine-readable verdict is src-tauri/tests/fixtures/continue_accuracy/release-report.json.

P6 Evidence-To-Answer Architecture

The implemented P6 production chain is:

AX/OCR/content/events
→ ordered evidence spans
→ salient user/agent/prior evidence
→ current task turn
→ scoped actions and semantic deltas
→ feedback/branch/workstream/open-loop eligibility
→ confidence and observation policy
→ local recap and optional validated model phrasing
→ target truth and evidence preview
→ interruption-recovery answer
→ React card and native island

The chain intentionally separates evidence extraction, semantic truth, wording, and opening policy. P6.02 emits smalltalk.task_turn_evidence.v1; P6.03 emits smalltalk.current_task_turn.v1 and persists task-turn evidence, relations, revisions, scoped actions, and semantic deltas. A task turn does not collapse lifecycle into one label: execution_state, current_actor, and waiting_on remain independent axes, with a separate relation to the prior task turn.

P6.04 persists feedback-policy evaluations rather than treating every old click or navigation as timeless approval. Provenance, task/session/workstream/target scope, freshness, decay, allowed effects, and per-target polarity constrain how feedback can suppress, cap, or support a candidate. P6.05 joins task turns to workstreams and evaluates semantic sources through smalltalk.semantic_eligibility.v1 and smalltalk.cross_layer_consistency.v1. The same semantic center must agree across the task turn, actions, workstream, open loop, branch, recap, and target; support or detour evidence does not silently become the primary task.

P6.06 produces smalltalk.continue_confidence.v2 and the compact smalltalk.continue_confidence_summary.v1. Surface identity, speaker/turn attribution, task identity, lifecycle state, semantic alignment, recap support, and target identity/openability/policy remain independently evidence-linked. Observation probes target a missing critical dimension and record success, successful-no-change, timeout, failure, privacy-blocked, or stale outcomes rather than manufacturing confidence.

P6.07 builds smalltalk.activity_recap_task_truth.v1 from smalltalk.activity_recap_inputs.v2. Deterministic local wording and the optional bounded model use the same task identity, lifecycle axes, semantic eligibility, claim-confidence caps, evidence handles, and target policy. Model wording is accepted only after local identity, temporal, workstream, source-eligibility, support, and target-policy validation; rejection returns a local fallback and cannot change candidate ranking or opening authority.

P6.08 applies smalltalk.direct_target_policy.v1 and emits smalltalk.continue_target_truth.v1, optional smalltalk.continue_evidence_preview.v1, and smalltalk.interruption_recovery_answer.v1. A direct target is an eligible, identified, policy-allowed, openable work object. An evidence preview is inspectable proof and must not masquerade as an open target. React presentation helpers and smalltalk.island_continue_state.v1 consume the same persisted decision semantics; native opening remains decision-id-only and runs strict live policy checks.

P6.01 and P6.09 provide the verification envelope through smalltalk.continue_accuracy_fixture.v1, smalltalk.continue_accuracy_report.v2, the frozen evaluation policy, deterministic first-divergence replay, privacy lint, corpus partitions, calibration/performance/model-parity results, and the explicit release gate. These contracts prove only the cases and denominators actually present in the report.

P6 Current Limitations And Release Status

P6 is implemented but is not release-complete. The important distinction is:

Implemented architecture: ordered role-aware evidence, first-class task turns, scoped feedback, semantic consistency, split confidence, bounded recap truth/model validation, target truth, evidence preview, a shared React/island answer contract, and the longitudinal evaluator are wired into the native Continue path.
Deterministically proven behavior: all seven committed synthetic Capture-button cases currently pass their P6 semantic checkpoints, presentation helpers have automated coverage, model validation preserves local identity in the covered fixtures, and frame fallback is not exposed as a public direct target in those cases.
Observed live behavior: the July 11 running app used the new P6 pipeline and persisted its new fields. This was not a stale frontend or stale backend build.
Release verdict: release_gate.passed is false. milestone_contract_passed describes the phase manifest only and is not authority to claim P6 ready.

The July 11 live Codex audit exposed the following primary-surface failures:

The captured Codex accessibility evidence contained the user's current draft, but the conversation-role resolver classified it as assistant history.
When no valid user span survived, the geometry fallback selected model-picker text as a user candidate. Its confidence was too low, after which the prior-boundary fallback promoted Approve for me.
The current task-turn resolver can use prior_boundary_sample as the provisional goal when no valid current user span exists. In this live case that turned old UI/history text into the apparent task.
Manual recap-model attempts correctly rejected packs that were inconsistent with the locally fixed task truth. That prevented an unsupported model rewrite, but the remaining local fallback was still unhelpful because its upstream task truth was wrong.
React currently accepts thin, fallback, or rejected recap content too readily for the primary headline instead of requiring a sufficiently supported current-task claim.
Surface quality and task/action quality are independently computed but insufficiently labeled in the card, so the first screen can show apparently contradictory Strong evidence and Thin evidence pills.
Generic detour and support explanations can dominate the first-screen answer even when they do not explain the user's actual current task.
The seven committed fixtures are synthetic and contain explicit role identifiers such as conversation-user-message and conversation-assistant-message. The live Codex accessibility tree did not expose those identifiers, so passing those fixtures does not demonstrate robust live Codex speaker recovery.

cloud_micro_inference means the bounded candidate router was used; it may merely return need_more_evidence. Wording provenance is therefore carried separately as wording_source, while task_understanding_source and target_selection_source describe their own stages. Manual Continue enables recap-model phrasing and startup/background refreshes keep recap phrasing local. The UI badge follows wording provenance, and React plus the island reject a weaker background result instead of allowing request source or completion order to overwrite a stronger manual answer.

The current machine-readable report remains intentionally closed because:

only seven synthetic cases exist, while release requires at least 100 independently human-reviewed cases;
the locked holdout is empty and locked-holdout evaluation has not been completed;
required calibration sample minimums are not met;
positive direct-target, no-clear-continuation, and supported-next-action denominators are missing or inadequate;
the measured model-off p95 in the latest completion audit exceeds the frozen regression budget;
manual native macOS interruption/recovery, direct-open, stale-open, and React/island parity QA is incomplete;
privacy and corpus review need the required independent human sign-off at release scale.

P6 therefore improves target safety and auditability, but it is not yet reliable enough at current-task recovery on the primary live Codex surface. A wrong or missing task must not be described as a P6 success merely because the direct target stayed null.

Task Truth v2.01 implements causal committed-typing attribution, shared control exclusion, history-only prior boundaries, a typed no_clear_current_task state through backend/React/island contracts, split task/wording/target provenance, and quality-dominant manual/background result adoption. Its live-shaped session-013 fixture intentionally omits synthetic conversation-role identifiers and proves that Approve for me cannot become the current task.

Task Truth v2.02 adds the privacy-safe live-corpus contract, local fixture builder, frozen evaluation policy, and three-path shadow evaluator. Task Truth v2.03 adds deterministic smalltalk.observation_packet.v2 packets, smalltalk.task_snapshot.v2, bounded checkpoints, task-only selection, and explicit-Continue shadow audits. Task Truth v2.04 adds the provider-neutral multimodal resolver, local claim verifier, conflict-only second pass, and deterministic answer wording.

Task Truth v2.05 adds the production authority policy (off, shadow, eligible, authoritative, rollback) and the versioned one-snapshot public answer. Target attachment remains subordinate to snapshot identity and strict-open policy. The authoritative React card, native island, adoption comparison, feedback path, and open command all use the same snapshot revision without filling missing fields from legacy P6. The direct target is persisted as a decision-scoped Task Truth contract, so a null public target cannot be revived by a legacy database target or open-time fallback. Exact-revision task feedback is isolated from global artifact/workstream feedback; it can remove a rejected field or promote a user-selected hypothesis without poisoning unrelated URLs, branches, or workstreams.

React and the island consume Task Truth only when the locked final report is configured and passes independent runtime validation. The runtime requires the exact schema and frozen policy, release-eligible corpus and holdout minimums, all semantic and per-surface metrics with non-zero denominators, Wilson confidence intervals, required slices, all 14 manual scenarios, a separate baseline-linked budget policy, performance/privacy evidence, and zero-tolerance counts. The pre-holdout baseline and holdout-enabled release evaluator are separate artifacts, and the budget policy must bind to the baseline's SHA-256 content identity. A hand-authored passed: true is insufficient. An attempted authoritative switch is reduced to eligible while the gate is closed, and every effective-state change is audited. The current final report remains false: there are five pending live-redacted development cases, zero independently reviewed release cases, zero locked holdout cases, and no frozen release-budget, performance/privacy, or manual macOS manifests. Production semantic authority therefore remains legacy P6 with Task Truth v2 shadow/eligible, not authoritative.

Evidence Anchors

A Continue answer must be explainable through anchors:

frame ids
event ids
action ids
episode ids
artifact ids
workstream ids
open-loop ids
branch-context ids
surface-snapshot ids
safe narrative-memory ids when prior context is explicitly used

The core decision keeps its frame/action/episode/artifact closure, while P5 evidence_spans can use the broader typed anchor set above for individual narrative claims. The UI should productize these into evidence previews and concise explanations. Raw ids belong in diagnostics.

P5 Activity Memory And Recap

P5 is an explanatory layer over the existing P1-P4 Continue decision. It does not replace candidate generation, scoring, branch policy, feedback suppression, quality gates, target validation, or strict open. Its job is to answer the part target selection alone cannot answer: what the user was doing, where they were doing it, what happened around that work, what state was left behind, and what can safely be suggested next.

Public Recap Contract

ContinueDecisionResult.activity_recap uses smalltalk.activity_recap.v1. Its public fields include:

primary_work_summary: concise evidence-backed description of the main activity.
primary_work_label: grounded activity label such as writing, planning, debugging, reviewing, reading documentation, or browsing files.
primary_where_summary: app, page, conversation, file, repository, or other surface label only when safely grounded.
activity_confidence: confidence in the activity narrative, independent of target availability.
target_confidence: confidence in the actionable return target, independent of activity understanding.
current_state: actively_working, recently_detoured, paused_after_progress, blocked, complete_or_idle, or unclear.
last_meaningful_state, unfinished_state, and next_action_summary.
recent_detours and supporting_context, each with typed roles, confidence, and evidence anchors.
why_this_target or why_no_safe_target.
missing_evidence, warnings, and per-claim evidence_spans.
generated_by: local, model_assisted, or fallback.
validation_status: valid, thin, rejected, or fallback.

Every public recap claim is sanitized, length-bounded, and required to retain an evidence span. Private locators, raw paths, URLs, internal ids, unsupported opaque handles, and ungrounded claims are removed rather than converted into plausible copy. A recap may confidently describe recent activity while keeping target_confidence low and both public target fields null.

Bounded Evidence Inputs

build_activity_recap_inputs runs after current-surface resolution and target selection. It creates smalltalk.activity_recap_inputs.v2 from already governed local facts:

current surface and its evidence quality, identity confidence, openability, and missing evidence;
selected workstream, selected candidate, return_target, and resume_work_target;
recent app-activity segments and classifications;
task actions and semantic moments;
open loops and workstream state snapshots;
branch contexts and promotion state;
weak-surface snapshots;
support evidence with explicit evidence-anchor ids;
bounded safe memory facts;
existing quality-gate, freshness, unresolved-current-work, and app-activity summaries.

Default caps are 12 activity segments, 40 actions, 30 semantic moments, 8 open loops, 8 workstream states, 8 branch contexts, 8 surface snapshots, 12 memory cells, and 12 support items. Selected-candidate relevance is bounded to seven days. Privacy-blocked, self/debug, unsafe-path, and non-claim-eligible facts are filtered or reduced to safe metadata. This layer does not make write-side capture calls and does not use raw-history fallback as its normal path.

Activity Timeline And Primary Work

stitch_activity_segments compresses graph evidence into at most six relevant segments. A segment can be classified as primary work, supporting work, detour, interruption, returned work, current-focus-only, or unclear. Adjacent evidence is merged only when identity and semantic boundaries agree; content changes, meaningful actions, returns, and branch transitions preserve boundaries.

Primary selection favors direct primary actions, selected-workstream continuity, stable work value, strong evidence, and explicit return evidence. A latest screen does not automatically become primary work. Brief Finder, Photos, search, documentation, terminal-output, messaging, and diagnostic surfaces remain detours or support unless fresh local promotion evidence shows that the branch became the unfinished task.

The grounded objective layer derives the activity kind and safe object/location terms from primary actions, snapshots, workstream evidence, and the stitched timeline. Browser chrome, generic app-only titles, sensitive fields, private locators, support/interrupt actions, and contradictory branch evidence are rejected as label sources. Where evidence supports only an app or broad surface, the recap stays broad instead of inventing a page, conversation, repository, or file.

Detours, State Left Behind, And Next Action

Detour recap keeps at most three public detours and three supporting-context items. Typed roles distinguish support, detour, interrupt, current-focus-only, promoted-primary, and unclear surfaces. Search/docs/messages/terminal/diagnostic branches can explain how the user got context or verified work without becoming the public return target.

Last-state synthesis combines grounded actions, open loops, workstream state, current-versus-primary continuity, snapshots, completion evidence, and blocker evidence. It can describe progress, an unfinished step, an unresolved blocker, completion/idle state, or uncertainty. why_this_target is emitted only for a locally eligible target; otherwise why_no_safe_target explains that activity may be understood even though no exact URL, path, page, thread, file, or other safely openable target is grounded.

The next action is constrained to evidence-backed continuation state. It cannot tell the user to open a target that local policy suppressed, promote a support branch, restore a rejected target, or manufacture a missing locator.

Optional Bounded Recap Phrasing

ContinueDecisionRequest.activity_recap_model_enabled defaults to false. When false, the deterministic local recap is returned without a recap model call. The React startup/background paths send false; explicit manual Continue, Refresh Continue, diagnostic rebuild, and explicit island Continue actions send true. This flag is separate from micro_inference_enabled, which controls candidate-bounded target inference.

When recap phrasing is explicitly enabled, the model receives smalltalk.activity_recap_model_pack.v2, containing only bounded current-surface facts, a primary segment, typed detours/support, the local recap seed, objective terms, safe next-action candidates, target policy, missing evidence, allowed term banks, and opaque evidence handles. The request uses Structured Outputs and a shorter bounded transport path. Raw screenshots, raw timelines, database dumps, typed characters, full clipboard text, private locators, and unfiltered history are excluded.

Local validation rejects or repairs output that:

uses unsupported terms or evidence handles;
changes activity or target confidence beyond local evidence;
describes an unknown detour or changes its role into primary work;
claims an openable target when local target policy says none exists;
emits a page, path, URL, id, state, next action, or target explanation not present in the bounded facts;
leaks internal/private text or exceeds public-copy limits.

Configuration, transport, parsing, or validation failure returns the local recap with explicit fallback/rejected status and warnings. Model-assisted phrasing never changes candidate identity, ranking, eligibility, openability, branch promotion, feedback suppression, return_target, or resume_work_target.

Cache, Persistence, Trace, Feedback, And Memory

Recap cache identity combines pipeline version, recap schema, model-enabled policy, effective model, and the full Continue evidence watermark. The evidence watermark covers meaningful frames/segments/classifications, branch contexts, surface snapshots, breadcrumbs, feedback, and decision-open events. Stable recomputation does not rewrite activity-classification identity merely to invalidate its own cache, and low-value mouse noise is not treated as a semantic recap change.

continue_decisions persists:

activity_recap_json;
activity_recap_detail_json containing smalltalk.activity_recap_decision_proof.v2;
activity_recap_watermark_hash;
activity_recap_policy_fingerprint;
activity_recap_model_requested.

A matching cache hit loads the exact stored recap and proof without rerunning synthesis. ContinueDecisionTrace.activity_recap exposes the same proof: bounded input summary, stitched timeline, work labels, detours, last-state derivation, model/fallback status, validation failures, and final recap.

Only valid medium/high-confidence recaps with a safe workstream or artifact scope and stable non-memory anchors can promote narrative cells: activity_workstream_summary, activity_primary_label, and activity_last_good_recap. Prior narrative memory may provide clearly labeled low-confidence context when fresh activity detail is thin, but it cannot create a target claim. Rejected, ignored, corrected, artifact-only, or ignored-workstream feedback downgrades matching recap memory into activity_recap_rejected; accepted feedback can strengthen matching narrative memory. A user next-step note stays neutral.

Narrative memory is hard-isolated from target selection. memory_candidate_links returns no candidate support/contradiction links for narrative memory types, so recap memory cannot affect ranking priors, candidate eligibility, target choice, alternatives, model candidate packs, openability, or strict open.

Audit Proof

Only explicit audit-enabled Continue actions write recap proof files. The asynchronous lean audit adds activity_recap/ with:

inputs_summary.json
stitched_timeline.json
work_labels.json
detours.json
last_state.json
final_recap.json
validation.json
model_pack.json
openai_request.redacted.json
raw_response.json
parsed_output.json
model_validation.json
fallback.json

When recap model phrasing was disabled or explicit audit retention was not requested, model-related files contain honest skipped/not-retained status rather than invented request or response data. This is implemented proof export, not a claim that every future P5 audit/eval roadmap metric already exists.

Default Bounded Micro-Inference

OpenAI micro-inference is the default Continue path. It is still bounded to local candidate ids, locally validated, and cache-aware. It does not receive broad raw history.

This target-selection inference is separate from P5 recap phrasing. micro_inference_enabled: true may choose among locally eligible candidates. Recap phrasing defaults off in the backend and stays local for startup/background refreshes, while the React manual Continue path explicitly enables it. Neither model path can override hard local safety gates.

The normal request is:

await invoke("get_continue_decision", {
  input: {
    mode: "normal",
    rebuild_layers: false,
    micro_inference_enabled: true,
    activity_recap_model_enabled: false,
    max_candidates_for_model: 5
  }
});

OpenAI key lookup:

process environment OPENAI_API_KEY
project .env

Model selection priority:

request model
SMALLTALK_CONTINUE_OPENAI_MODEL
SMALLTALK_OPENAI_MODEL
OPENAI_MODEL
default gpt-4.1-mini

The model receives a compact candidate pack only. It contains:

current focus facts
top workstreams
top continuation candidates
candidate ids generated locally
target artifact ids
target kinds and titles
booleans for URL/path availability
local score components
last meaningful action summaries
unresolved-state reasons
evidence frame/action/episode ids
missing evidence notes
artifact role map
short manual breadcrumbs
factual unresolved current-work state when present
bounded current-surface/enrichment quality and freshness facts
support evidence that is visible to reasoning but not selectable as a return target
feedback policy and branch-promotion eligibility already computed locally

The model does not receive:

raw screenshots by default
raw timelines
raw database dumps
raw typed characters
full clipboard text
unredacted URLs
unredacted file paths
frames excluded by privacy policy

Structured output fields:

selected_candidate_id
selected_workstream_id
intent_label
next_action
reason
confidence: low, medium, or high
uncertainty_notes

The model output is validated locally. The validator rejects output when:

selected candidate id was not supplied locally
selected workstream id does not match the selected candidate
selected candidate was not sent to the model
output mentions unsupported URLs or paths
output leaks internal candidate, workstream, artifact, frame, or fallback identifiers into handoff copy
next_action is empty, too long, or incompatible with candidate semantics
high confidence is returned for thin evidence
a branch/support target is promoted without a strong local candidate
the candidate was suppressed by feedback, excluded by branch policy, or omitted from the filtered model pack
a stale target is selected over fresh strong/medium current work

If the API fails, the key is missing, parsing fails, or validation fails, the decision source becomes local_fallback and the local scorer result is returned.

Fallback decisions are still cached when their evidence watermark and inference policy match the next normal request. This matters because default micro-inference should not repeatedly attempt network/model work when the same local evidence already produced a validated local fallback.

The decision layer keeps semantic task understanding separate from target safety. When a model answer exists but there is no human-readable, strictly supported return target, Smalltalk shows the model answer and changes the action to evidence inspection. It does not suppress the paid model response merely because direct opening is unavailable. Internal target metadata remains hidden, and no unsupported app, page, URL, or file is opened.

Micro-inference cannot override the local safety gates. A model choice does not create promotion evidence, restore a feedback-suppressed candidate, make a thin snapshot openable, or turn diagnostic/support evidence into a public target.

Feedback And Breadcrumbs

Continue feedback has two forms:

Inferred feedback.
Explicit UI feedback.

The inferred command is:

infer_continue_feedback

It can infer:

accepted: user returned to suggested target and stayed or acted there
rejected: user opened target but quickly left with no meaningful action
ignored: no target activity appeared inside the observation window
corrected: user chose another artifact shortly after Continue
auto_resumed: user naturally returned to the workstream without using the suggestion

The explicit command is:

record_continue_feedback

It supports:

accepted
rejected
ignored
corrected
artifact_only_evidence
ignored_workstream
user_next_step_note

Explicit feedback is deduped through deterministic ids. Notes are capped at 500 characters in the backend. The frontend currently caps breadcrumb text to 240 characters before sending.

Feedback is aggregated through the versioned feedback_obedience.v1 policy rather than a count-only rule. The reducer weighs explicit and inferred events, distinguishes soft score caps from hard suppression, records the last negative event, and looks for fresh reconfirming evidence after that event. Repeated rejection/ignore signals can make a target ineligible for primary selection; artifact_only_evidence keeps an artifact as evidence while suppressing public promotion.

continue_decision_open_events records the lifecycle needed to infer feedback and invalidate caches: decision/candidate/workstream/artifact ids, source, whether an open was attempted/allowed/succeeded, strategy, timestamp, and bounded warnings. It intentionally does not store raw URL or path text. A matured open without confirming activity, or new explicit feedback, changes the feedback/open watermark and prevents a stale cached decision from being reused.

Negative feedback from the main card or island triggers a rebuilt Continue answer instead of leaving the rejected target visible. The same live feedback state is checked again during strict open.

Breadcrumbs are stored through:

add_continue_breadcrumb

A breadcrumb is a short local-only note on a workstream. It can be included in later bounded candidate packs. It must not be treated as an external artifact.

Workstream Detail

The frontend loads workstream detail only in diagnostics:

await invoke("get_continue_workstream_detail", {
  input: {
    workstream_id: selectedWorkstreamId,
    decision_id: continueDecision?.decision_id || null
  }
});

The detail result contains:

workstream summary
artifact details
episode details
candidate details
latest decision summary
feedback events
breadcrumbs
evidence anchors

This is excellent for debugging but too dense for the default product surface.

Continue Eval

The eval command is:

run_continue_eval

Default fixture invocation:

await invoke("run_continue_eval", {
  evalFilePath: null
});

Custom fixture invocation:

await invoke("run_continue_eval", {
  evalFilePath: "/absolute/path/to/continue-eval.json"
});

Eval report fields:

schema
case count
target artifact correctness
Recall@k
MRR
current-focus false-positive rate
hallucinated artifact count
model validation fallback rate
last-state and next-action specificity
support-branch handling, false-promotion rate, origin recall, and promoted-branch precision
thin-evidence honesty, truthful thin-mode rate, and no-clear-continuation correctness
current-focus/return clarity, open-loop recovery, source provenance, and quality-gate correctness
cache freshness, fresh-current-work retention, and stale-target suppression
feedback suppression exposure, corrected-artifact preference, stale feedback-cache hits, model feedback violations, and suppressed-target open attempts
weak-surface enrichment attempt/success/quality, truthful thin rendering, candidate recall, stale-URL false positives, fake-open targets, missing-evidence rendering, and privacy violations
P1 feedback-gate and P2 support-gate regression counters
island bypass, legacy-primary-route, missing-decision-id open, suppressed-target open, main-card disagreement, and valid-open success counters
per-case results

Eval belongs in Developer diagnostics.

Stop-Time Resume Query Path

The stop-time cloud resume path is separate from Continue.

When the user pauses local memory through stop_capture, the backend:

Stops runtime.
Marks session stopped.
Refreshes status/counts.
Builds a bounded resume-query bundle.
Writes generated artifacts under resume_query_exports/.
Returns StopCaptureOutput.

StopCaptureOutput includes:

status
session
export
resume_query
preview

The resume-query schema in capture_core/resume_dossier.rs is:

smalltalk.resume_query.v2

Default resume-query policy:

max_json_chars: 25,000
max_model_images: 12
max_episode_cards: 8

Requested JSON and image limits are capped at those defaults.

Resume-query bundles can include recent_surface_context. This is context-only evidence extracted from rejected browser-chrome anchors, especially tab-strip titles. These labels can explain that another browser tab was briefly visible, but they are explicitly not resume anchors and should not become the return target.

The stop-time path is useful for bounded cloud reasoning, but it is not the core Continue engine.

Cloud Resume Path

run_cloud_resume is the older stop/resume-query model path. It should not be confused with get_continue_decision.

Cloud resume:

builds or reuses a bounded resume-query bundle
makes an OpenAI Responses API call when configured
can request targeted follow-up evidence when the model says need_more_evidence
validates anchor contracts locally
persists source/provenance
requires a real response_id for a trusted cloud result

Trusted cloud output has:

source: "cloud"
non-empty response_id

Local fallback output is explicitly source: "local_fallback".

The user has previously treated fake cloud success as a correctness failure, so the UI and docs must preserve provenance.

Open Resume Point

open_resume_point still supports diagnostic compatibility inputs, but product opens are now source-aware and policy-gated. It can resolve targets from:

Continue decision id
cloud resume output path
session id
current frame id
target frame id

The primary React card sends continue_decision_id, optional displayed target_artifact_id, source: "desktop_continue_card", and strict_continue_target: true. The island sends source: "island_primary", strict_continue_target: true, and only a persisted decision id.

Opening can use:

browser URL when allowed and openable
document path when allowed and openable
diagnostic frame fallback only when the caller explicitly uses a diagnostic source with diagnostic_allowed: true
Smalltalk focus fallback when opening is blocked

Open result includes:

strategy
opened URL/path flags
warnings

The UI should not fabricate targets. If the backend only provides a frame anchor, the UI should inspect that frame rather than inventing a URL or path.

Strict Continue open resolves the exact target associated with the persisted decision, proves that artifact belongs to the decision/workstream, rechecks current feedback suppression, requires a real direct locator for public opening, and refuses stale legacy fallback fields. island_primary fails closed unless all strict conditions hold. Every attempt is recorded best-effort in continue_decision_open_events without raw URL/path text.

Privacy And Security

Privacy boundaries:

Do not store raw typed characters.
Do not store full clipboard text.
Store keyboard categories, counts, and commit signals instead.
Store clipboard metadata, hashes, and provenance instead of content.
Apply exclusion and privacy rules before storing heavy frames.
Mark or skip sensitive frames.
Preserve privacy_status.
Store sensitive regions and actions taken.
Exclude never_send_to_ai frames from model-facing exports.
Redact raw URLs and file paths from bounded micro-inference packs.
Do not commit .env, API keys, SQLite DBs, screenshots, capture exports, or resume-query exports.

Safe AI export means derived/redacted evidence plus audit rows, not raw screenshot dumps.

continue_outputs/ is different. It is a private developer proof bundle created only for explicit Continue actions. The default audit is lean and includes the decision trace, final decision/handoff, quality and freshness gates, candidate/feedback/branch state, current-surface and weak-surface diagnostics, model/cache/copy validation, selected evidence closure, manifest, integrity metadata, and explain.md. It does not copy the whole SQLite database, every table, or every frame by default.

Full SQLite snapshots, streaming raw-table NDJSON, schema dumps, and all-frame capture archives are opt-in through SMALLTALK_CONTINUE_AUDIT_FULL_RAW=1 or an effective mode containing full_raw. Audit output may still contain sensitive local evidence and paths; it is generated private output and must never be committed or uploaded accidentally.

Cleanup And Retention

The current app includes developer-facing cleanup controls:

Preview cleanup
Apply cleanup
Dev reset

cleanup_local_memory accepts:

include_debug_exports
vacuum
dry_run

It returns:

diagnostics
dry-run flag
candidate frame count
protected frame count
deleted frame count
deleted event-row count
deleted snapshot file count
reclaimed bytes
summary

Cleanup is explicit; the capture worker does not silently delete history merely because a session reaches one hour. Preview is the default.

Frame cleanup candidates are:

frames older than seven days, in bounded batches;
old Smalltalk self-captures;
low-value typing_pause, scroll_stop, click, accessibility_change, event_burst, and idle frames beyond the newest 400.

Event cleanup candidates are:

Smalltalk self-events;
low-value scroll/AX/accessibility rows beyond the newest 5,000.

Cleanup protects frame ids referenced by candidates, task-action first/last/strongest evidence, semantic moments, episodes, artifact observations, explicit/manual/hotkey evidence, and decision/high-value privacy markers. It deletes dependent relational rows and JPEG assets, removes orphan snapshots, prunes old unreferenced decisions/candidates, checkpoints the WAL, and optionally vacuums. Recent decisions keep at least the newest 100 decision rows even beyond the 24-hour age boundary; selected candidates and feedback links are preserved consistently.

The 512 MB snapshot constant is a pressure gate for new non-important heavy captures, not automatic deletion. If the directory is over budget, important/manual/error evidence can still be accepted and the user can preview/apply cleanup later.

dev_reset_local_memory is stronger. It stops runtime, clears live frames/events/derived Continue rows/snapshots, and can clear debug exports.

Frontend Runtime Behavior

Important frontend state includes:

status
continueMemory
memoryDiagnostics
cleanupResult
continueDecision
continueDecisionFrameCount
continueDecisionUpdatedAt
workstreams
selectedWorkstreamId
workstreamDetail
feedbackStatus
evalReport
selectedFrame
frameDetail
timeline
imageData
evidenceOpen
diagnosticsOpen

The P5 recap is carried inside continueDecision.activity_recap; React does not run a separate narrative request. usableActivityRecap accepts only smalltalk.activity_recap.v1, and product-copy filters reject private locators, URLs, paths, internal ids, semantic/debug labels, and scorer terminology before rendering.

The app refreshes status immediately on mount and then polls:

every 1.5 seconds while local memory is running
every 6 seconds while stopped

When running and diagnostics are open, it also refreshes:

Continue memory
search results
timeline
workstreams

When a capture-frame event arrives, the frontend refreshes status and Continue memory. If no frame is selected, it selects the new frame. When diagnostics are open, it refreshes workstreams.

The frontend auto-runs Continue once when:

no Continue decision exists
no busy action is active
frame count is greater than zero
the auto-continue guard has not already fired

The UI marks a decision stale when the live frame count exceeds the frame count used when the decision was made. This is only a freshness hint; event-only evidence can also affect backend cache decisions.

The production freshness path is broader than frame count. React maintains a ContinueEvidenceSnapshot/ContinueFreshness signature covering frames, events/signals, Continue-memory counts, and island/backend update information. Refreshes are debounced, guarded against overlap, and avoid recomputing the same stale signature. The backend remains authoritative through its evidence watermark, boundary revisions, surface snapshots, feedback watermark, and open watermark.

Startup/background Continue calls use writeAudit: false. Main Continue, Refresh Continue, diagnostic Rebuild Continue, and explicit island Continue can request writeAudit: true. After the main card receives a decision, it synchronizes the island using the existing decision id with allow_refresh: false rather than launching duplicate work.

When a usable recap exists, the primary card presents:

You were working on plus the grounded activity summary.
Where when the app/page/file/conversation surface is safe and supported.
Recent context for bounded detour or support summaries.
State left behind.
Next.
The separate safe return-target block or Exact return target missing.
Independent activity and target confidence.
Why-this-target/no-safe-target reasoning and missing evidence.

The recap may improve the headline and state explanation in openable, thin-current-work, enriched-but-not-openable, older-context, and no-clear-continuation modes. It does not change whether Continue here is available. That action still requires a non-empty decision id, non-no_clear_continuation output, an openable non-support target, and the backend's strict-open policy.

Diagnostics UI

Developer diagnostics include:

Workstream list.
Next-step note form.
Workstream detail.
Local memory storage metrics.
Cleanup controls.
Rebuild Continue button.
Search captured evidence.
Capture health strip.
Continue eval panel.
Evidence timeline.
Raw event stream.
Frame screenshot viewer.
Overlay controls for content units, OCR, AX, and privacy.
Verification drawer.
Text/events/context/path tabs.

Diagnostics are intentionally detailed but should not be mistaken for the first-run product experience.

Session Island

The macOS floating island is now a typed Continue-first consumer. Rust exposes IslandContinueState (smalltalk.island_continue_state.v1) with:

display state;
decision id;
current focus/activity;
activity label, summary, where, and state;
separate activity and target confidence labels;
bounded recent-context summary;
selected workstream title;
return and resume-work target summaries;
next action and confidence label;
missing evidence, warnings, and suppression reasons;
typed available_actions.

Swift decodes this nested DTO and dispatches typed actions such as refresh_continue, open_continue_target, mark_wrong_target, mark_not_useful, inspect_evidence, open_smalltalk, start_local_memory, and capture_evidence_now. Legacy cloud/session/trail/native-resume routes remain diagnostic-only and cannot supply the island's primary target/open behavior.

The island obtains state from the same get_continue_decision backend contract as the main card or from a fresh remembered decision. Its primary open requires source = island_primary, strict_continue_target = true, and a non-empty continue_decision_id; legacy path/session/frame fallbacks are rejected before resolution. Feedback uses existing feedback kinds with source = island_primary. Frame/event, feedback, and open watermarks invalidate remembered island state.

Rust maps the same persisted recap into island fields; Swift displays the compact hierarchy without recomputing or widening claims. Thin or no-clear states can still show useful activity memory, state, context, and missing evidence, but they cannot expose the primary open action unless the same decision is currently safe and openable in the main Continue contract.

P4 no-bypass coverage writes sanitized decision/island_continue_audit.json metadata and tracks island bypass, legacy primary route, open-without-decision-id, suppressed-target open, main-card disagreement, and valid-open counters. The repeatable manual checklist is docs/p4-island-no-bypass-manual-qa.md.

Current Implementation Truth

What is real now:

Native desktop app is the active lane.
Continue schema version 9 exists in SQLite, including ordered evidence, salient turn evidence, task turns and lifecycle, scoped feedback evaluations, task/workstream membership, semantic eligibility/consistency, and persisted decision truth/proof fields.
P6.02-P6.08 are integrated into the production get_continue_decision chain, not only into an offline evaluator.
ContinueDecisionResult carries current_task_turn, semantic consistency and direct-target policy, target truth, evidence preview, split confidence, the recap, and the final interruption-recovery answer.
The longitudinal accuracy evaluator, privacy lint, release report, and explicit P6 release verdict are implemented. The authoritative verdict is currently false.
Capture events, sparse heavy frames, attributed text, weak-surface snapshots, artifacts/actions/semantic moments/episodes/workstreams/open loops/branches/candidates/decisions/opens/feedback are persisted locally.
Continue can run without stopping local memory.
Normal Continue mode can reuse cached decisions.
Default micro-inference can fall back locally and reuse that cached fallback when evidence has not changed.
Every new Continue result carries smalltalk.activity_recap.v1, including independent activity/target confidence, current state, evidence-backed narrative fields, detours/support, missing evidence, and per-claim anchors.
Activity recap inputs are bounded and privacy-filtered; recap generation does not send broad raw history or use raw-history fallback as the normal path.
Deterministic P5 stitching distinguishes primary work, support, detours, interruptions, returns, and current-focus-only surfaces before inferring a work label or state.
Activity/location labels are designed and deterministically tested to use eligible actions, snapshots, segments, workstream evidence, and evidence spans. Generic chrome, private locators, support-only actions, and unsafe internal labels have rejection paths. The live Codex audit proves that role/region resolution can still feed the wrong upstream task truth, so this must not be generalized into a real-surface accuracy claim.
Last-state synthesis can distinguish active work, recent detour, pause after progress, blocker, complete/idle, and unclear state while keeping no-safe-target copy honest.
Recap model phrasing exists and defaults off in the backend. React manual Continue/Refresh/Rebuild and explicit island Continue actions enable it; startup/background recaps stay local. When enabled it uses a bounded Structured Outputs pack, local term/handle/target-policy validation, and deterministic fallback.
Recap cache identity includes evidence plus recap/model policy. Matching cache hits reuse the exact stored recap and proof without rerunning synthesis.
Valid grounded recap memory can be promoted and adjusted by feedback, but narrative memory is isolated from candidate ranking, target eligibility, alternatives, openability, and strict open.
React and the native island consume the same persisted recap and display compact activity, location, context, state, next action, confidence, and missing-evidence copy.
Explicit audit-enabled Continue outputs include an activity_recap/ proof directory; background/startup decisions still do not create audit bundles.
Continue handoff copy is persisted and filtered so internal candidate/workstream/artifact/frame ids do not become user-facing product text.
Only explicit audit-enabled Continue actions schedule a continue_outputs/ bundle. Startup/background calls do not write one. Folder names begin with the resolved capture session label.
Audit generation runs asynchronously and is lean/proof-first by default; full raw archives are opt-in.
Continue micro-inference audit events record the candidate pack, OpenAI request body without secrets, raw response, parsed output, validation result, failures, and fallback reason.
Diagnostic rebuild can force semantic layer rebuild.
The UI opens Continue targets by continue_decision_id.
Correction feedback and next-step breadcrumbs are persisted.
Local memory diagnostics expose storage size, row counts, budgets, skip counters, and cache counters.
Heavy capture budgets and duplicate/self-capture skipping exist.
Long sessions retain lightweight event-only evidence without forcing a screenshot per event, and cleanup can prune excess low-value event rows while preserving semantic/decision evidence.
Event-only evidence is a first-class part of current-surface resolution, semantic moments, cache freshness, and weak-surface recovery.
Visible local signal counts are recent-windowed and capped.
Stop-time resume-query preserves rejected browser tab-strip titles as context-only evidence.
Stop-time resume-query bundles still exist and are generated artifacts.
Cloud resume is distinct from Continue.
Support branches, stale openable targets, feedback-suppressed targets, and thin weak surfaces are hard-gated before public selection/open.
The floating island is aligned to the backend Continue contract and fails closed on legacy primary-open bypasses.
SQLite uses WAL, a 30-second busy timeout, read-only polling connections, and serialized Continue decision work to reduce long-session lock contention.
Legacy database migration now adds continue_task_actions.task_turn_id before creating its index, and the UI no longer reports that SQLite schema failure as an Accessibility permission problem.
The diagnostics panel still exposes too much internal architecture when opened.

What should not be claimed:

Do not claim the browser extension is the active MVP.
Do not claim Stop Session is required for Continue.
Do not claim cloud resume is the primary product engine.
Do not claim screenshots are the only memory.
Do not claim model output is trusted without validation.
Do not claim recap model phrasing runs by default; activity_recap_model_enabled defaults to false.
Do not claim activity confidence implies target confidence or openability.
Do not claim P5 activity memory changes candidate ranking or can restore/promote a target.
Do not claim a page, thread, file, repository, or URL is known when only an app or broad surface is grounded.
Do not claim a brief latest detour is the primary task merely because it is the current focus.
Do not claim storage is lightweight unless diagnostics prove it.
Do not claim a one-hour session stores a continuous replay, a fixed frame count, or automatic hour-boundary cleanup.
Do not claim the 512 MB snapshot budget is a hard disk cap; it gates new non-important heavy captures and requires explicit cleanup to reclaim space.
Do not claim the island can open a legacy session/frame/path target as a primary Continue action.
Do not claim P6 is release-ready because seven synthetic cases pass or because milestone_contract_passed is true; only release_gate.passed is authoritative.
Do not claim current-task recovery is reliable on real Codex accessibility evidence; the July 11 audit demonstrated a speaker/region fallback failure.
Do not claim AI-assisted proves the visible recap was model-written; candidate micro-inference and recap wording have separate provenance.
Do not claim local fallback is necessarily useful when upstream task-turn extraction is wrong.
Background refresh preserves a stronger manual answer unless the challenger has causally newer evidence without a task, state, target, confidence, or wording-provenance downgrade.

Product Copy Rules For Future LLMs

Use these product words:

Continue
local memory
current focus
what you were working on
where
recent context
state left behind
return target
workstream
activity confidence
target confidence
evidence
next action
confidence
missing evidence
correction

Avoid making these words first-class product copy on the primary screen:

session
recorder
frame id
action id
episode id
artifact id
raw event stream
bundle
scorer
candidate score
resume query
cloud resume
FTS
SQLite

Those terms belong in diagnostics and technical docs.

Implementation Checklist For Future Changes

When changing Smalltalk, check these boundaries:

Does the primary screen still produce one continuation answer?
Does Continue work without Stop Session?
Are current_focus, activity_recap, return_target, and resume_work_target still separate?
Are branch/support surfaces prevented from becoming default return targets unless evidence supports it?
Does every answer have frame/action/episode/artifact anchors?
Does thin evidence remain explicit?
Are raw typed characters and full clipboard text still excluded?
Are generated exports ignored?
Are model calls bounded to candidate ids and locally validated?
Are diagnostics kept out of the first-run product surface?
Are storage budgets, cleanup, and skip counters preserved?
Are tests added for deterministic classifier/scoring/storage changes?
Does every public activity-recap claim retain a valid evidence span?
Can activity memory still explain thin work without creating an openable target?
Are recap memory and optional model phrasing still unable to affect ranking, eligibility, promotion, or strict open?
Does the current task turn come from valid current-user evidence rather than a control label or unsupported prior boundary?
Are execution state, current actor, and waiting-on kept separate?
Do task, surface, and target confidence remain separately labeled and separately capped?
Does the React card distinguish decision routing provenance from recap wording provenance?
Can a startup/background refresh replace a manual answer only when it is semantically at least as strong?
Is release_gate.passed, rather than a phase milestone flag or synthetic-case count, used for the P6 release claim?

Recommended Verification For A Product Change

Run Rust formatting, compile checks, and the full deterministic suite:

cd src-tauri
cargo fmt --check
cargo check
cargo test

Run the React build and presentation-contract tests:

cd /Users/bhaskarpandit/Documents/smalltalk
npm run build
npm run test:continue-presentation

Regenerate the P6 accuracy report without unlocking the holdout:

cd /Users/bhaskarpandit/Documents/smalltalk/src-tauri
cargo run --bin continue_accuracy_eval -- \
  --output tests/fixtures/continue_accuracy/release-report.json \
  --repeat 3

Run the committed-fixture contract and privacy-lint checks explicitly when editing the corpus or fixture schema:

cd /Users/bhaskarpandit/Documents/smalltalk/src-tauri
cargo test continuation::accuracy_fixture::tests
cargo test continuation::accuracy_eval::tests

Do not add --allow-locked-holdout during normal development or threshold tuning. The locked holdout is release evidence, not a debugging set.

For live interruption-recovery behavior:

cd /Users/bhaskarpandit/Documents/smalltalk
npm run tauri dev

Follow docs/phases/p6-task-turn-accuracy/p6-08-manual-qa-results.md and record the native results rather than inferring them from unit tests. Exercise manual Continue, startup/background refresh, interruption and return, no-safe-target evidence preview, positive direct open, stale/suppressed open, model-on/model-off identity, and React/island parity. During the run, inspect the live SQLite database with the read-only queries in the storage section; an empty or old continue_outputs/ folder does not mean the app made no decisions.

For a documentation-only refresh, also run:

cd /Users/bhaskarpandit/Documents/smalltalk
git diff --check

Manual QA should verify:

The first screen reads as Continue, not recorder/debug.
Continue runs while local memory is active.
Continue also returns a thin-evidence answer when evidence is insufficient.
Current focus and return target are visibly separate.
Activity summary, location, recent context, state left behind, and next action are shown only when grounded.
Activity and target confidence can differ without collapsing into one confidence claim.
Surface, task, and target quality do not appear as contradictory unlabeled confidence claims.
The displayed task is supported by current user/task-turn evidence, including on the real Codex accessibility shape.
The provenance display distinguishes candidate-routing AI from recap-wording AI.
A background result does not silently downgrade a stronger manual result.
A brief Finder/search/docs/messages/terminal detour remains context unless local evidence promotes it.
Useful activity memory can appear when the exact safe target is missing, without showing Continue here.
The primary target can be opened or falls back to evidence inspection.
Wrong-target correction records feedback.
Alternatives can be selected without inventing missing URLs.
Diagnostics are hidden until opened.
Memory cleanup preview does not delete protected decision-linked frames.
Delete/reset clears live UI state after clearing backend state.
The P6 release label remains closed unless the generated report has release_gate.passed: true and all manual evidence is complete.

Glossary

Term	Meaning
Continue	Main product action that returns the user to the next actionable point.
Local memory	Local evidence store built from events, frames, text, app context, and derived semantic rows.
Frame	Heavy captured evidence row, usually with screenshot and text sources.
Signal	Lightweight event or evidence count that may not include a screenshot.
Artifact	Stable local work object such as tab, doc, conversation, editor, terminal, message thread, or PDF.
Observation	Evidence that an artifact appeared in a frame or event-backed moment.
Task action	Local inferred action such as editing, searching, encountering an error, or returning to origin.
Episode	Adjacent actions grouped by continuity and boundary reasons.
Workstream	Durable cluster of related episodes and artifacts.
Candidate	A scored possible continuation target.
Decision	Persisted Continue answer with source, confidence, validation, warnings, and anchors.
Current focus	Latest factual observed screen/artifact.
Activity recap	Evidence-backed explanation of the primary work, where it happened, recent context, state left behind, next action, uncertainty, and target rationale.
Activity confidence	Confidence that Smalltalk understands what the user was doing; it does not imply a safe return target.
Target confidence	Confidence in the exact actionable return target; it can remain low or none even when activity confidence is higher.
Activity segment	Bounded interval of related app/surface evidence used to distinguish primary work, support, detours, interruptions, returns, and current-focus-only activity.
Detour	Recent surface that explains context switching but is not primary work unless fresh local promotion evidence says otherwise.
Supporting context	Search, docs, terminal output, messages, diagnostics, or other evidence used around the primary work without becoming its default target.
Return target	Where Smalltalk thinks the user should go back.
Resume work target	The actionable target inside the workstream.
Breadcrumb	Manual local next-step note attached to a workstream.
Feedback event	Explicit or inferred signal about whether a Continue decision was useful.
Resume-query bundle	Stop-time bounded export for cloud resume, separate from native Continue.
Cloud resume	Older OpenAI path over resume-query bundles.
Micro-inference	Optional candidate-bounded OpenAI ranking/phrasing layer for Continue.
Recap model phrasing	Separately gated, disabled-by-default model rewrite of bounded local recap facts; locally validated and unable to alter target policy.
Evidence anchor	Frame/action/episode/artifact id that explains a Continue result.
Evidence span	Per-claim mapping from public recap copy to bounded local frame, event, action, episode, workstream, open-loop, branch, snapshot, or memory anchors.

Model-First Task Truth Production Contract

Smalltalk's production semantic authority is the cloud multimodal Task Truth path. Local code still captures and redacts evidence, orders the observation stream, verifies model claims, maintains task-thread revisions, persists results, and validates return targets. It does not invent the user's task, current subtask, relationship to earlier work, unfinished state, or next action.

The only valid semantic sources for the first-screen Continue answer are:

cloud_multimodal_model for a provider result that passed local evidence verification;
human_correction for a correction scoped to the exact task snapshot or hypothesis;
unresolved when the provider is unavailable, the request is blocked, the evidence is insufficient, the response is invalid, or verification rejects the central task claim.

A provider failure must never fall back to a title-derived label such as "Browsing", "Editing", or "Reviewing output". The app may show the observed surface inside Why this answer?, but that surface is evidence rather than inferred task truth. An older verified answer may remain visible while a manual refresh is running only when it is clearly the previous result; it cannot absorb newer local surface fields.

Every public model-first answer is one atomic revision. The session, observation packet, task thread and revision, selected hypothesis, model request and response, evidence watermark, and correction state travel together. React and the native island consume that same answer. A direct Continue here action is attached only after local validation proves that the return anchor belongs to the same task-thread revision. Understanding the task does not imply that a safe target exists.

Production authority is fail-closed behind the model-first release report. Deterministic tests and a successful provider transport call are necessary but not sufficient. The gate also requires independently reviewed live decision boundaries, the locked application-level holdout, non-zero denominators for every required slice and surface family, provider-failure honesty, latency/cost/privacy measurements, and the manual macOS scenarios. Missing human review or a zero denominator keeps passed = false; it never re-enables legacy local semantic authority.

Always-On Runtime Boundary

Smalltalk's runtime now treats pressure as a finite resource. Native event metadata crosses three bounded queues with capacity reserved for app, window, error, permission, and commit boundaries. Scroll, Accessibility changes, repeated keys, ordinary character-category keys, and repeated clicks can coalesce only within documented surface and time windows. Coalescing preserves timestamps, counts, surface identity, and privacy-safe aggregate diagnostics. A capture-loop turn persists at most 32 events and then returns to Stop, pending capture, and idle-capture checks.

Database creation and migration run at an explicit database-generation boundary. Startup initializes the selected database once. Reset or file replacement invalidates that generation and initializes the replacement. Ordinary event, frame, status, counter, and Continue operations open configured connections without running schema discovery or Data Definition Language (DDL) statements. The capture worker owns its event-ingest connection and uses short atomic batches with bounded busy retry.

Continue, capture, audit, and maintenance use one finite workload policy. Manual Continue supersedes queued background work. A cancelled background result cannot replace the later manual result on the product surface. Full audits are manual-only and use one worker with one pending request. Maintenance is single-flight and chunked. Neither system may create a thread per request.

Status is a lightweight product snapshot. It reads maintained session counters and a latest-frame projection that omits OCR text, Accessibility text, Accessibility trees, URLs, document paths, and image paths. Heavy evidence is loaded only through explicit evidence commands. Event-driven updates are primary; the slow heartbeat is recovery insurance.

The runtime policy and developer harness are documented in docs/runtime-stability-harness.md and versioned in docs/runtime-stability-policy-v1.json. Automated pressure tests are not a substitute for the required live soak matrix. Until every live scenario passes, the truthful release state remains automated and synthetic proof complete; live always-on proof pending.

FilesExpand file tree

PRODUCT.md

Latest commit

History