Skip to content

v0.11.0: facts layer, reviewed writes, delta sync, no silently dropped sources - #23

Merged
renezander030 merged 10 commits into
mainfrom
release-0.11.0
Aug 22, 2026
Merged

v0.11.0: facts layer, reviewed writes, delta sync, no silently dropped sources#23
renezander030 merged 10 commits into
mainfrom
release-0.11.0

Conversation

@renezander030

Copy link
Copy Markdown
Owner

v0.11.0 in one branch: a facts layer, enforced write approval, honest partial results everywhere, delta-capable cache sync, and portable state. Ten commits, one per concern, in review order below. The mcp package is untouched beyond re-exports — the CLI remains the growth surface.

What's in it

Facts layer — ats kg (new). Durable subject–predicate–object knowledge beside the tasks, in an embedded append-only JSONL log (no graph server, nothing to operate). Single writer by construction: agents propose facts and retractions; only ratify writes the store, after a human approves — every fact records proposer, ratifier, source, and temporal validity. ats kg ask is deterministic lexical scoring with provenance (zero LLM); retraction closes the validity interval instead of deleting. ats kg export --cypher emits a LadybugDB/Kùzu-dialect load script. Proposals ride the same review queue as gated task writes, so ats review works on them unchanged.

Reviewed writes. intent.approvalRequired and security.approvalRequiredFor were declarative only; the CLI now enforces them. Guarded update/complete/delete stage into the review queue (the update path reuses the read it already makes for undo before-images, so no extra fetch); ATS_REVIEW_ALL=1 gates everything including creates. ats review apply executes through the normal adapter path with the approver recorded in the ledger — reviewed writes stay undoable. A failed apply keeps the item approved with its error recorded.

No silently dropped sources. The three remaining omission paths named in the README now reach warnings/degraded: composite fallback per-project failures, composite children whose native search errors (bubbled namespaced via __searchWarnings, which core reads after the native branch), and TickTick project fetches across its corpus loader, search, hybrid pool, and vector sync. The TickTick loader also stops caching a known-partial corpus — it previously cached whatever survived, serving the subset as complete and healthy for the whole TTL.

Fusion identity. Composite task ids are namespaced <backend>:<taskId> like project ids, closing the documented cross-backend id collision in RRF fusion. Routing accepts namespaced and raw ids both ways, so ids copied from find output resolve unchanged.

Cache sync + delta hook. ats cache sync|status|clear now falls back to Core's corpus cache for every adapter (it previously errored without an adapter cache extension, while doctor reported staleness with no remedy). New optional adapter hook bulkFetchDelta({cursor, since}): changes apply as whole-item replacements — never a field merge — with the cursor persisted in the cache. Backends without a changes API (TickTick's open API) keep full refresh.

Completed history in retrieval. New optional contract method listCompletedTasks(); ats find --include-completed appends completed items per query without ever writing them into the shared cache. TickTick maps its existing completed-tasks support onto the contract; composite unions children and names the ones that cannot answer; adapters without the method degrade the result with an explicit warning.

Concurrency safety. A shared lock/atomic-write module (extracted from the event spool's proven pattern) now guards the corpus cache (atomic replace under lock — a torn cache was possible), the ledger append (before-image lines can exceed the atomic-append size), and the whole undo critical section (two concurrent undos of one action can no longer double-apply). Verified with a multi-process lost-update test.

Trust boundaries. Composite children can be marked "trust": "public"; writes routed there are screened against configured redaction patterns and blocked with the rule named — never silently stripped. Invalid patterns fail loudly at load. Both READMEs state the honest scope: this guards ATS's own composite write path, not arbitrary data movement.

Portable state. ats state export|import bundles ledger, undo images, review queue, event checkpoint/spool, usage log, caches, and index metadata. Credentials are never bundled (whitelist registry), and import writes only to the local registry's paths — a crafted bundle cannot redirect a write. Also answers hosted-deployment ephemerality.

Hygiene + onboarding + backfill. ats dedup apply turns clusters into typed links (optionally closing duplicates) through the normal ledgered, review-gated path; ats garden reports stale-but-active tasks with per-task archive commands, detection only. ats agent-setup emits the CLAUDE.md/AGENTS.md policy block from live config. ats sync vector --all drains the embedding backfill in capped rounds, stopping without forward progress.

Verification

  • Full gate green locally on Node 22: lint, PII + claims checks, 250+ unit tests across packages (36 new), intent/taskmaster/beads proofs, progress benchmark. The pre-commit hook ran the same gate on every commit.
  • New coverage includes: multi-process lost-update test for the lock module; composite collision, warnings, redaction, and completed-history fan-out; corpus delta replace/remove/cursor semantics; review-queue lifecycle incl. failed-apply retry; kg propose→ratify→ask→retract round-trip; state-bundle import path-authority (bundle-supplied paths are ignored).
  • Live CLI smoke on this machine: ats agent-setup against the real config, and the full kg propose → review approve → kg ratify → kg ask loop end-to-end.
  • README "Tradeoffs and limits" updated where behavior changed, so the claims checker holds the docs to the new reality.

Deliberately not in this branch

  • No version bump / tag / publish — the release commit (lockstep bump across packages, CHANGELOG date, tag, npm publish in dependency order) follows your review, per repo convention. The CHANGELOG section is drafted and marked Unreleased.
  • No MCP feature work. New capabilities are CLI-first; the MCP server only gains what falls out of shared core exports.
  • No native LadybugDB driver. The store is embedded JSONL; graph-engine users get a tested Cypher export instead of an untestable native binding. The driver seam is a clean follow-up if wanted.

Parallel agents are the normal case, and three state files could interleave:
the corpus cache was replaced with a plain write (torn reads possible), the
action ledger appended without a lock (before-image lines can exceed the
size the OS appends atomically), and undo's read-verify-append allowed two
concurrent undos of the same action to both pass the already-undone check
and double-apply against the backend.

fs-lock.js centralizes the spool's proven O_EXCL + stale-steal lock pattern
plus a temp-and-rename atomic replace; the spool, cache, ledger append, and
the whole undo critical section now go through it. Lock is advisory and
writer-side only — reads stay lock-free.
…espaced

Three admitted omission paths now reach warnings/degraded: a project that
fails inside the composite fallback fetch, a child whose native search
errors (or reports its own unreadable projects via __searchWarnings, which
core now reads after the native branch), and TickTick project fetches in
the adapter's own corpus loader, search, hybrid keyword pool, and vector
sync. The TickTick loader also stops caching a known-partial corpus — it
previously cached whatever survived, serving the subset as complete and
healthy for the whole TTL.

Composite task ids are now namespaced <backend>:<taskId> like project ids,
so two backends emitting the same raw id can no longer merge into one
result in RRF fusion. Routing (getTask/updateTask/urlFor) accepts both
namespaced and raw ids, so ids copied from find output resolve unchanged.
… drain

'ats cache sync' previously errored on any adapter without a centralized
cache extension — the doctor could report the corpus cache stale while no
command existed to refresh it. The cache subcommands now fall back to
Core's corpus cache for every adapter: status, sync (cron-friendly), clear.

Core gains syncCorpusCache() with an optional adapter hook, bulkFetchDelta
({cursor, since}), for backends that can answer what-changed-since: changed
tasks apply as whole-item replacements over the prior corpus and removedIds
delete — never a field merge, which is how stale-cache corruption starts.
The adapter's cursor persists inside the cache file. Backends without a
changes API (TickTick's open API) skip the hook and get a full refresh; a
full fetch with failing sources is reported and never cached as complete.

'ats sync vector --all' drains the embedding backfill in rounds of the
per-run cap instead of leaving the tail to repeated manual --max runs; it
stops on a round with no forward progress rather than spinning.
…ontract

Retrospective queries ('what was actually done') could not be answered from
find: the corpus only ever held active tasks, and completed history existed
solely as a TickTick-specific __ext command invisible to other adapters.

The adapter contract gains optional listCompletedTasks({since, until,
projectIds}); 'ats find --include-completed' appends its results per query —
never into the shared corpus cache — each carrying status:'completed' (branch
projections now preserve status). The TickTick adapter maps its existing
/task/completed support onto the contract and its own find loader does the
same cache-bypassing append. The composite unions children and records the
ones that cannot answer; an adapter without the method degrades the result
with 'completed history is not supported' instead of silently answering from
active tasks only.
The approval metadata was declarative only: intent.approvalRequired and
security.approvalRequiredFor existed on tasks, but an agent write went
straight to the backend regardless. Update/complete/delete now check the
target's own metadata (the update path reuses the read it already makes for
undo before-images) and stage guarded writes into a review queue instead;
ATS_REVIEW_ALL=1 stages every write, which also covers creates.

ats review list/show/approve/reject/apply runs the queue. Apply executes
through the normal adapter write path with the approver recorded in the
action ledger (approvals field), so reviewed writes stay undoable; a failed
apply keeps the item approved with its error, never silently lost. The
queue store is generic (kind-tagged) so later propose-review flows share
the same mechanics.
ats state bundles the derived state that previously had no move/backup path
— action ledger with undo before-images, review queue, event checkpoint and
spool, usage log, caches, index metadata — into one JSON document and
restores it on another install. Two hard boundaries: credentials are never
bundled (the registry is a whitelist of state files; adapter configs and
.env files are not in it), and import writes only to the local registry's
paths — a crafted bundle cannot redirect a write. This is also the
persistence answer for ephemeral hosted deployments: export before teardown,
import after.

ats agent-setup emits the paste-able CLAUDE.md/AGENTS.md policy block that
makes an agent use the CLI correctly — generated from the live configuration
(active adapter and its origin, wiki project), so the block always matches
the install it runs against: retrieval-first with degraded-result honesty,
patch-semantics writes, typed links and intent, the review-gate stop rule,
deep links via ats url.
…edup apply + garden

Trust boundary: composite children can be marked trust:'public'; a
createTask/updateTask routed to a public child is screened against the
configured redaction patterns and BLOCKED with the matching rule named —
never silently stripped — so content picked up from a private backend
cannot flow into a public one through ATS unnoticed. An invalid pattern
fails loudly at config load: a protective rule must never drop silently.
Scope stated honestly in both READMEs: this guards the composite's own
write path, it is not general DLP.

ats dedup apply turns a detected duplicate cluster into typed links
(--keep/--dupes, supersedes by default, conflicts-with optional) and can
close the duplicates — all through the normal write path, so links and
closures are ledgered, undoable, and subject to the review gate (a guarded
duplicate stages instead of closing). Task refs split on the LAST slash so
namespaced project ids like github:owner/repo survive.

ats garden sweeps the corpus for active tasks untouched past a threshold
(default 60d) and prints a per-task archive command. Detection only, by
design — bulk hygiene that silently mutates is the exact failure mode the
sweep exists to prevent.
Agents accumulate durable plain-language knowledge that outlives any task.
ats kg stores it as subject-predicate-object facts with temporal validity
and provenance in an embedded, serverless append-only log (no graph server,
nothing to operate; travels with ats state export).

Three properties by design: single writer — nothing writes the store except
ratify, agents only PROPOSE (kind kg.fact in the shared review queue, so
ats review list/approve/reject already work on facts) and every fact records
proposer, ratifier, and source; facts are events — retraction closes the
validity interval instead of deleting, so what-did-we-believe-then stays
answerable; zero-LLM reads — ats kg ask is deterministic lexical scoring
(subject > object > predicate, phrase bonus, newest-first ties) with full
provenance in the answer.

ats kg export --cypher emits a load script for embedded Cypher engines
(LadybugDB / Kùzu dialect: node table Entity, rel table FACT), and the
agent-setup policy block teaches agents the propose-not-write discipline.
…es stated

The task layer is record-based by design and cannot hold knowledge written
from any source about mixed subjects into one space; the README now says up
top that the kg layer exists for exactly that, and states the recommended
pairing explicitly: Graphiti as the graph database server, LadybugDB as the
embedded graph database (Cypher export loads it directly; JSON export feeds
a Graphiti ingest pipeline).
@renezander030
renezander030 merged commit e9f7635 into main Aug 22, 2026
3 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant