v0.11.0: facts layer, reviewed writes, delta sync, no silently dropped sources - #23
Merged
Conversation
Parallel agents are the normal case, and three state files could interleave: the corpus cache was replaced with a plain write (torn reads possible), the action ledger appended without a lock (before-image lines can exceed the size the OS appends atomically), and undo's read-verify-append allowed two concurrent undos of the same action to both pass the already-undone check and double-apply against the backend. fs-lock.js centralizes the spool's proven O_EXCL + stale-steal lock pattern plus a temp-and-rename atomic replace; the spool, cache, ledger append, and the whole undo critical section now go through it. Lock is advisory and writer-side only — reads stay lock-free.
…espaced Three admitted omission paths now reach warnings/degraded: a project that fails inside the composite fallback fetch, a child whose native search errors (or reports its own unreadable projects via __searchWarnings, which core now reads after the native branch), and TickTick project fetches in the adapter's own corpus loader, search, hybrid keyword pool, and vector sync. The TickTick loader also stops caching a known-partial corpus — it previously cached whatever survived, serving the subset as complete and healthy for the whole TTL. Composite task ids are now namespaced <backend>:<taskId> like project ids, so two backends emitting the same raw id can no longer merge into one result in RRF fusion. Routing (getTask/updateTask/urlFor) accepts both namespaced and raw ids, so ids copied from find output resolve unchanged.
… drain
'ats cache sync' previously errored on any adapter without a centralized
cache extension — the doctor could report the corpus cache stale while no
command existed to refresh it. The cache subcommands now fall back to
Core's corpus cache for every adapter: status, sync (cron-friendly), clear.
Core gains syncCorpusCache() with an optional adapter hook, bulkFetchDelta
({cursor, since}), for backends that can answer what-changed-since: changed
tasks apply as whole-item replacements over the prior corpus and removedIds
delete — never a field merge, which is how stale-cache corruption starts.
The adapter's cursor persists inside the cache file. Backends without a
changes API (TickTick's open API) skip the hook and get a full refresh; a
full fetch with failing sources is reported and never cached as complete.
'ats sync vector --all' drains the embedding backfill in rounds of the
per-run cap instead of leaving the tail to repeated manual --max runs; it
stops on a round with no forward progress rather than spinning.
…ontract
Retrospective queries ('what was actually done') could not be answered from
find: the corpus only ever held active tasks, and completed history existed
solely as a TickTick-specific __ext command invisible to other adapters.
The adapter contract gains optional listCompletedTasks({since, until,
projectIds}); 'ats find --include-completed' appends its results per query —
never into the shared corpus cache — each carrying status:'completed' (branch
projections now preserve status). The TickTick adapter maps its existing
/task/completed support onto the contract and its own find loader does the
same cache-bypassing append. The composite unions children and records the
ones that cannot answer; an adapter without the method degrades the result
with 'completed history is not supported' instead of silently answering from
active tasks only.
The approval metadata was declarative only: intent.approvalRequired and security.approvalRequiredFor existed on tasks, but an agent write went straight to the backend regardless. Update/complete/delete now check the target's own metadata (the update path reuses the read it already makes for undo before-images) and stage guarded writes into a review queue instead; ATS_REVIEW_ALL=1 stages every write, which also covers creates. ats review list/show/approve/reject/apply runs the queue. Apply executes through the normal adapter write path with the approver recorded in the action ledger (approvals field), so reviewed writes stay undoable; a failed apply keeps the item approved with its error, never silently lost. The queue store is generic (kind-tagged) so later propose-review flows share the same mechanics.
ats state bundles the derived state that previously had no move/backup path — action ledger with undo before-images, review queue, event checkpoint and spool, usage log, caches, index metadata — into one JSON document and restores it on another install. Two hard boundaries: credentials are never bundled (the registry is a whitelist of state files; adapter configs and .env files are not in it), and import writes only to the local registry's paths — a crafted bundle cannot redirect a write. This is also the persistence answer for ephemeral hosted deployments: export before teardown, import after. ats agent-setup emits the paste-able CLAUDE.md/AGENTS.md policy block that makes an agent use the CLI correctly — generated from the live configuration (active adapter and its origin, wiki project), so the block always matches the install it runs against: retrieval-first with degraded-result honesty, patch-semantics writes, typed links and intent, the review-gate stop rule, deep links via ats url.
…edup apply + garden Trust boundary: composite children can be marked trust:'public'; a createTask/updateTask routed to a public child is screened against the configured redaction patterns and BLOCKED with the matching rule named — never silently stripped — so content picked up from a private backend cannot flow into a public one through ATS unnoticed. An invalid pattern fails loudly at config load: a protective rule must never drop silently. Scope stated honestly in both READMEs: this guards the composite's own write path, it is not general DLP. ats dedup apply turns a detected duplicate cluster into typed links (--keep/--dupes, supersedes by default, conflicts-with optional) and can close the duplicates — all through the normal write path, so links and closures are ledgered, undoable, and subject to the review gate (a guarded duplicate stages instead of closing). Task refs split on the LAST slash so namespaced project ids like github:owner/repo survive. ats garden sweeps the corpus for active tasks untouched past a threshold (default 60d) and prints a per-task archive command. Detection only, by design — bulk hygiene that silently mutates is the exact failure mode the sweep exists to prevent.
Agents accumulate durable plain-language knowledge that outlives any task. ats kg stores it as subject-predicate-object facts with temporal validity and provenance in an embedded, serverless append-only log (no graph server, nothing to operate; travels with ats state export). Three properties by design: single writer — nothing writes the store except ratify, agents only PROPOSE (kind kg.fact in the shared review queue, so ats review list/approve/reject already work on facts) and every fact records proposer, ratifier, and source; facts are events — retraction closes the validity interval instead of deleting, so what-did-we-believe-then stays answerable; zero-LLM reads — ats kg ask is deterministic lexical scoring (subject > object > predicate, phrase bonus, newest-first ties) with full provenance in the answer. ats kg export --cypher emits a load script for embedded Cypher engines (LadybugDB / Kùzu dialect: node table Entity, rel table FACT), and the agent-setup policy block teaches agents the propose-not-write discipline.
…es stated The task layer is record-based by design and cannot hold knowledge written from any source about mixed subjects into one space; the README now says up top that the kg layer exists for exactly that, and states the recommended pairing explicitly: Graphiti as the graph database server, LadybugDB as the embedded graph database (Cypher export loads it directly; JSON export feeds a Graphiti ingest pipeline).
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
v0.11.0 in one branch: a facts layer, enforced write approval, honest partial results everywhere, delta-capable cache sync, and portable state. Ten commits, one per concern, in review order below. The
mcppackage is untouched beyond re-exports — the CLI remains the growth surface.What's in it
Facts layer —
ats kg(new). Durable subject–predicate–object knowledge beside the tasks, in an embedded append-only JSONL log (no graph server, nothing to operate). Single writer by construction: agentsproposefacts and retractions; onlyratifywrites the store, after a human approves — every fact records proposer, ratifier, source, and temporal validity.ats kg askis deterministic lexical scoring with provenance (zero LLM); retraction closes the validity interval instead of deleting.ats kg export --cypheremits a LadybugDB/Kùzu-dialect load script. Proposals ride the same review queue as gated task writes, soats reviewworks on them unchanged.Reviewed writes.
intent.approvalRequiredandsecurity.approvalRequiredForwere declarative only; the CLI now enforces them. Guarded update/complete/delete stage into the review queue (the update path reuses the read it already makes for undo before-images, so no extra fetch);ATS_REVIEW_ALL=1gates everything including creates.ats review applyexecutes through the normal adapter path with the approver recorded in the ledger — reviewed writes stay undoable. A failed apply keeps the item approved with its error recorded.No silently dropped sources. The three remaining omission paths named in the README now reach
warnings/degraded: composite fallback per-project failures, composite children whose native search errors (bubbled namespaced via__searchWarnings, which core reads after the native branch), and TickTick project fetches across its corpus loader, search, hybrid pool, and vector sync. The TickTick loader also stops caching a known-partial corpus — it previously cached whatever survived, serving the subset as complete and healthy for the whole TTL.Fusion identity. Composite task ids are namespaced
<backend>:<taskId>like project ids, closing the documented cross-backend id collision in RRF fusion. Routing accepts namespaced and raw ids both ways, so ids copied fromfindoutput resolve unchanged.Cache sync + delta hook.
ats cache sync|status|clearnow falls back to Core's corpus cache for every adapter (it previously errored without an adapter cache extension, whiledoctorreported staleness with no remedy). New optional adapter hookbulkFetchDelta({cursor, since}): changes apply as whole-item replacements — never a field merge — with the cursor persisted in the cache. Backends without a changes API (TickTick's open API) keep full refresh.Completed history in retrieval. New optional contract method
listCompletedTasks();ats find --include-completedappends completed items per query without ever writing them into the shared cache. TickTick maps its existing completed-tasks support onto the contract; composite unions children and names the ones that cannot answer; adapters without the method degrade the result with an explicit warning.Concurrency safety. A shared lock/atomic-write module (extracted from the event spool's proven pattern) now guards the corpus cache (atomic replace under lock — a torn cache was possible), the ledger append (before-image lines can exceed the atomic-append size), and the whole undo critical section (two concurrent undos of one action can no longer double-apply). Verified with a multi-process lost-update test.
Trust boundaries. Composite children can be marked
"trust": "public"; writes routed there are screened against configured redaction patterns and blocked with the rule named — never silently stripped. Invalid patterns fail loudly at load. Both READMEs state the honest scope: this guards ATS's own composite write path, not arbitrary data movement.Portable state.
ats state export|importbundles ledger, undo images, review queue, event checkpoint/spool, usage log, caches, and index metadata. Credentials are never bundled (whitelist registry), and import writes only to the local registry's paths — a crafted bundle cannot redirect a write. Also answers hosted-deployment ephemerality.Hygiene + onboarding + backfill.
ats dedup applyturns clusters into typed links (optionally closing duplicates) through the normal ledgered, review-gated path;ats gardenreports stale-but-active tasks with per-task archive commands, detection only.ats agent-setupemits the CLAUDE.md/AGENTS.md policy block from live config.ats sync vector --alldrains the embedding backfill in capped rounds, stopping without forward progress.Verification
ats agent-setupagainst the real config, and the fullkg propose → review approve → kg ratify → kg askloop end-to-end.Deliberately not in this branch