diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index e68042d4..d68783b9 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -110,6 +110,10 @@ jobs: # #1131: the embedder-less profile (cloud/OpenAI embedders only, no # onnx) must keep compiling — the embedding cache is backend-agnostic. - run: cargo check -p uteke-core --no-default-features --features vecq + # #1168: dual-engine profile (usearch + vecq compiled together, runtime + # selection) must keep compiling and passing the engine-switch tests. + - run: cargo test -p uteke-core --features "usearch,vecq" --lib vector_engine + - run: cargo test -p uteke-core --features "usearch,vecq" --lib memory::vector docs-check: name: API Docs Fresh diff --git a/.github/workflows/cla-check.yml b/.github/workflows/cla-check.yml index 98820d72..f838384f 100644 --- a/.github/workflows/cla-check.yml +++ b/.github/workflows/cla-check.yml @@ -12,7 +12,7 @@ permissions: jobs: cla-check: runs-on: ubuntu-latest - if: "!contains(fromJSON('[\"app/dependabot\", \"app/renovate\", \"github-actions[bot]\"]'), github.event.pull_request.user.login)" + if: "!contains(fromJSON('[\"dependabot[bot]\", \"renovate[bot]\", \"github-actions[bot]\", \"app/dependabot\", \"app/renovate\"]'), github.event.pull_request.user.login)" steps: - name: Fetch & check CLA signature id: check @@ -48,7 +48,7 @@ jobs: - name: Comment on PR (unsigned only) if: steps.check.outputs.signed != 'true' - uses: actions/github-script@v7 + uses: actions/github-script@v9 with: script: | const author = '${{ github.event.pull_request.user.login }}'; @@ -97,7 +97,7 @@ jobs: } - name: Set commit status - uses: actions/github-script@v7 + uses: actions/github-script@v9 with: script: | const signed = '${{ steps.check.outputs.signed }}' === 'true'; diff --git a/CHANGELOG.md b/CHANGELOG.md index 39737d25..b4d762eb 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,5 +1,34 @@ # Changelog +## [0.17.0] — 2026-09-06 + +Minor release. Theme: **inspectable, trustworthy memory** — explain recall on +every surface, auditable conflict resolution with a measurable payoff, honest +graphs, pagination metadata, and a dual-engine vector layer. + +### Added + +- **Contradiction benchmark segment (#1172, phase 3)** — `benchmarks/longmemeval/contradiction_segment.py`: 40-topic active-store segment measuring conflict-resolution quality end-to-end. Baseline (both facts active) vs resolved (superseded): fusion winner@1 0.975 → 1.000, stale@5 0.825 → 0.000. Published in `benchmarks/longmemeval/RESULTS.md`. Also adds `uteke supersede [--reason]` — CLI surface parity for supersession (previously MCP/HTTP only). + +- **`/list` pagination metadata (#1188)** — `POST /list` accepts `"include_meta": true` to respond with an envelope `{memories, total, has_more, next_offset}` (`next_offset` is `null` on the last page) so clients no longer blind-paginate with 100-row guesses. The default response is unchanged (bare array) — existing clients are untouched; `include_meta` is ignored in `at` (point-in-time) mode, which stays a bare array. + +- **Explain recall (#1160)** — `explain` mode on every recall surface shows WHY each memory ranked where it did: vector similarity and rank, FTS rank, RRF score with per-channel fusion contributions, and jaccard/salience/recency/graph boost deltas. Surfaces: `uteke recall "…" --explain` (human-readable, combine with `--json` for machine output), `POST /recall` with `"explain": true` (memory-only — combined with `search_type`/`at`/`before`/`after` returns 400), and the `explain` flag on the MCP `uteke_recall` tool. The explanation path replays the active strategy's exact pipeline (same channel depths, RRF constants, and boost order) while bypassing the recall cache, so the explanation always matches the returned results; fts5 explanation works without an embedder, other strategies embed the query once (~50 ms, same as a cold recall). + +- **Contradiction resolution ledger + undo (#1172, phase 2)** — supersessions are now a first-class, auditable ledger instead of a side effect: `Uteke::contradiction_resolutions(namespace, limit)` lists superseded-but-not-restored memories (winner, reason, timestamp via the deprecation row), `Uteke::undo_supersession(id)` restores a retired memory, removes the supersession edge pair, and records a `supersession_undone` event on both sides (only memories carrying a live `superseded_by` edge can be undone — the undo is itself auditable). Ledger membership is edge-driven (deprecated row + `superseded_by` edge), the same predicate undo resolves against, and re-superseding an already-deprecated memory refreshes the stored reason/timestamp so the ledger always names the current winner. Surfaces: `GET /contradictions?namespace=&limit=`, `POST /contradictions/undo` (`{id}`; 404 when nothing to undo), `uteke contradictions list|undo`, and MCP `uteke_contradictions` / `uteke_contradictions_undo`. Fixed in the process: the no-namespace ledger query bound its limit parameter to a nonexistent placeholder (`?2`) and failed at runtime — caught by the new MCP roundtrip test. + +- **Provenance data model (#1172, phase 1)** — schema v18 (additive): `memories.source_hash` records the SHA-256 of content at write time (tamper evidence — audits recompute it against live content), and `timeline_events.actor`/`evidence_json` record who performed an event and what evidence supports it. New `Uteke::provenance(id)` returns the full report (provenance fields, trust tier, hash comparison, event chain) — exposed as `GET /provenance?id=`, `uteke provenance `, and the `uteke_provenance` MCP tool. + +### Fixed + +- **Graph data returned stale nodes (#1189)** — `GET /graph` without a namespace returned every `graph_nodes` row raw, including nodes whose parent memory had been forgotten or deprecated; with soft-delete the store accumulated stale nodes on every conflict resolution. Memory-linked nodes are now filtered by liveness (memory exists and `deprecated = 0`) in every `graph_data` path, edges touching removed nodes are dropped, and `stats` counts the filtered graph. +- **Memory graph nodes labeled with raw UUIDs (#1187)** — `ensure_node_for_memory` now labels new memory nodes with a readable content preview (first 60 chars of the memory) instead of the raw memory UUID, and upgrades legacy UUID-labeled rows in place on next access. Entity nodes are unaffected. + +- **Namespace management API (#1181)** — namespaces are a derived view, now with sanctioned ops: `PUT /memory` accepts `namespace` (move a memory — plain column update, no re-embed), `POST /namespaces/rename` (`{from, to}`; existing target = merge, returns `{from, to, moved, target_existed}`), and `POST /namespaces/delete` with an explicit strategy for its memories: `refuse` (default — 409 while any memory references the name), `merge` (move all memories to `target`, the name vanishes), or `deprecate` (soft-delete — restorable via promote, never hard-deleted). `GET /namespaces?with_counts=true` now adds `active`/`deprecated` breakdown fields (`count` stays the total). CLI parity: `uteke namespace move|rename|delete` (delete requires `--confirm`). MCP parity: `uteke_namespace_rename`, `uteke_namespace_delete`, and `namespace` field on `uteke_update`. + +### Fixed + +- **`POST /graph/edge` always returned 500 for valid memory IDs (#1180)** — the handler validated `source`/`target` as memory IDs but inserted them directly into `graph_edges`, whose foreign keys point at `graph_nodes(id)`. Memory IDs are now resolved to their linked graph node (or a node is ensured automatically) before insertion. `DELETE /graph/edge` accepts memory IDs or graph node IDs the same way, and its documented query params are corrected to `?source=...&target=...`. `POST /graph/edge` now responds with `{ok, source_node, target_node}` so clients can track the created nodes. + ## [0.16.0] — 2026-08-28 Minor release. One theme: retrieval quality that ships by default. diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 04c55171..fe173bc3 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -272,4 +272,28 @@ See [CODE_OF_CONDUCT.md](CODE_OF_CONDUCT.md). ## License -By contributing you agree your work is licensed under [Apache-2.0](LICENSE). No CLA required. +By contributing you agree your work is licensed under [Apache-2.0](LICENSE). + +## CLA + +All contributions (code, docs, tests, configuration) require a signed +Contributor License Agreement before a pull request can be merged: + +- 📋 **Individual?** → [Sign the Individual CLA](https://codecoradev.github.io/cla/?type=individual) +- 🏢 **Contributing on behalf of a company?** → [Sign the Corporate CLA](https://codecoradev.github.io/cla/?type=corporate) + +The CLA is a license agreement, not a copyright assignment — you keep +ownership of your work. Signing takes a couple of minutes and is stored +in the [codecoradev/.github](https://github.com/codecoradev/.github) +repository; a bot checks it automatically on every pull request. + +## Contributions are unpaid + +Contributing to this project is **voluntary and unpaid**. There is no +compensation, payment, bounty, or financial reward of any kind for +contributions — now or in the future. You contribute on your own time, +at your own discretion, because you want to improve the project. + +If any paid-contribution program is ever introduced, it will be announced +explicitly and this document will be updated. Until then, assume every +contribution is volunteer work under the Apache-2.0 license terms above. diff --git a/Cargo.lock b/Cargo.lock index ed0034e2..5265ae6e 100644 --- a/Cargo.lock +++ b/Cargo.lock @@ -727,12 +727,13 @@ checksum = "26b73573e6edcd2af0cdf47bd6cb58f0b3839491263c314eaad1ccf24430e1de" [[package]] name = "flate2" -version = "1.1.9" +version = "1.1.10" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "843fba2746e448b37e26a819579957415c8cef339bf08564fe8b7ddbd959573c" +checksum = "6e634e2e0ebac1ee034020da1ca582e17ffe4e0f5e985823721e168928136dcb" dependencies = [ "crc32fast", "miniz_oxide", + "zlib-rs", ] [[package]] @@ -1348,9 +1349,9 @@ checksum = "68354c5c6bd36d73ff3feceb05efa59b6acb7626617f4962be322a825e61f79a" [[package]] name = "miniz_oxide" -version = "0.8.9" +version = "0.9.1" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "1fa76a2c86f704bdb222d66965fb3d63269ce38518b83cb0575fca855ebb6316" +checksum = "b63fbc4a50860e98e7b2aa7804ded1db5cbc3aff9193adaff57a6931bf7c4b4c" dependencies = [ "adler2", "simd-adler32", @@ -2755,9 +2756,9 @@ checksum = "daf8dba3b7eb870caf1ddeed7bc9d2a049f3cfdfae7cb521b087cc33ae4c49da" [[package]] name = "usearch" -version = "2.26.0" +version = "2.26.1" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "1cd7f672d20412962c457b11c858c6c5aecb949808a5345a95e1d671112bcf72" +checksum = "884002b91c545f43b67e9eadbef70834502ad14c32df1ce8d72bdb56ab7d8be3" dependencies = [ "cxx", "cxx-build", @@ -2766,7 +2767,7 @@ dependencies = [ [[package]] name = "uteke-cli" -version = "0.15.0" +version = "0.17.0" dependencies = [ "chrono", "clap", @@ -2792,7 +2793,7 @@ dependencies = [ [[package]] name = "uteke-core" -version = "0.15.0" +version = "0.17.0" dependencies = [ "chrono", "dirs", @@ -2816,7 +2817,7 @@ dependencies = [ [[package]] name = "uteke-mcp" -version = "0.15.0" +version = "0.17.0" dependencies = [ "chrono", "dirs", @@ -2828,7 +2829,7 @@ dependencies = [ [[package]] name = "uteke-server" -version = "0.15.0" +version = "0.17.0" dependencies = [ "chrono", "ctrlc", @@ -2859,9 +2860,9 @@ checksum = "06abde3611657adf66d383f00b093d7faecc7fa57071cce2578660c9f1010821" [[package]] name = "uuid" -version = "1.24.0" +version = "1.26.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "bf3923a6f5c4c6382e0b653c4117f48d631ea17f38ed86e2a828e6f7412f5239" +checksum = "b5772d71c9be8a8a6ac2117d949c5b224c1b72241bb611d9a3012edcf8af7812" dependencies = [ "getrandom 0.4.3", "js-sys", @@ -2882,9 +2883,9 @@ checksum = "accd4ea62f7bb7a82fe23066fb0957d48ef677f6eeb8215f372f52e48bb32426" [[package]] name = "vecq-core" -version = "0.1.1" +version = "0.3.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "07959c480b6d1d1d0d80eb92fa322271d6c2f1ae69529d6b0fbefc6dd4c88693" +checksum = "543ddd2e748c26c39bd8ca5ab9c43eb01ba0ca13750a1428b08e8c7dc0d9def7" [[package]] name = "version_check" @@ -3002,9 +3003,9 @@ dependencies = [ [[package]] name = "which" -version = "8.0.5" +version = "8.0.6" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "8f3ef584124b911bcc3875c2f1472e80f24361ceb789bd1c62b3e9a3df9ff43c" +checksum = "bae2f2b2b816647a1cab1acc91f5bd20812d53cb344382635ec2181940c8034f" dependencies = [ "libc", ] @@ -3312,6 +3313,12 @@ dependencies = [ "syn 2.0.119", ] +[[package]] +name = "zlib-rs" +version = "0.6.7" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "34b31d188d9d685a4f9c7b46d6e36631b07058d2cfe190267adce54dc230bf12" + [[package]] name = "zmij" version = "1.0.23" diff --git a/Cargo.toml b/Cargo.toml index fd6ddf59..22394222 100644 --- a/Cargo.toml +++ b/Cargo.toml @@ -9,7 +9,7 @@ members = [ ] [workspace.package] -version = "0.16.0" +version = "0.17.0" edition = "2024" license = "Apache-2.0" repository = "https://github.com/codecoradev/uteke" diff --git a/README.md b/README.md index 48d2b4e6..16033f6a 100644 --- a/README.md +++ b/README.md @@ -142,7 +142,13 @@ Every AI tool forgets. Context windows fill up, sessions end, and your AI starts | **Recall latency (10K memories)** | **42ms** P50, 50ms P95 | Flat from 100 to 10K memories (HNSW O(log N)) | | **Insert throughput** | 6-22 ops/s | CPU-bound (ONNX embedding inference) | | **Storage per memory** | ~10KB | SQLite + HNSW, scales linearly | -| **LongMemEval Recall@5** | **0.946** | Full 500Q validation, zero-config fusion default (R@10 0.977), EmbeddingGemma Q4 | +| **LongMemEval-S recall_any@5** | **98.2%** | Full 500Q validation, zero-config fusion default (v0.16.0) — the metric competitor benchmarks publish | +| LongMemEval-S recall_all@10 | 95.4% | Strict: every gold session in top-10 | +| LongMemEval-S strict recall_all@5 | 88.0% | Every gold session in top-5 (mathematical ceiling 99.4%) | + +![uteke vs published systems on LongMemEval-S](docs/assets/longmemeval-comparison.jpg) + +> **Don't trust our benchmark — run your own.** We re-ran 108 of the 500 published questions on a 4-core ARM desktop (different CPU architecture from the published Modal x86 run, same v0.16.0 binary and harness): **107/108 produced identical per-question rankings**. The single difference was an adjacent-rank near-tie, both runs retrieved the identical top-10 session set, one gold session swapped ranks 5-6. Details in [RESULTS.md](benchmarks/longmemeval/RESULTS.md). Full benchmarks: `uteke bench --counts 100,1000,10000 --json` · [Benchmark details](docs/benchmarks.md) · [LongMemEval results](benchmarks/longmemeval/RESULTS.md) — fusion default: R@5 0.946 / R@10 0.977 on the full 500Q validation set (v0.16.0) diff --git a/benchmarks/longmemeval/RESULTS.md b/benchmarks/longmemeval/RESULTS.md index a5664877..e5540c98 100644 --- a/benchmarks/longmemeval/RESULTS.md +++ b/benchmarks/longmemeval/RESULTS.md @@ -5,6 +5,28 @@ --- +## Independent Reproduction (2026-09-01) + +The published 500Q run was produced on Modal x86. To test whether the result depends on that infrastructure, a 108-question subset (pref: 30, kupd: 78) was re-run locally on a 4-core ARM desktop (Oracle Ampere A1, aarch64) — same v0.16.0 binary, same harness (`run_eval.py`), different CPU architecture. + +| Subset | Questions | Identical per-question rankings | R@5 published | R@5 re-run | +|---|---|---|---|---| +| pref | 30 | 30/30 | 96.7% | 96.7% | +| kupd | 78 | 77/78 | 100.0% | 99.4% | +| **Total** | **108** | **107/108** | — | — | + +**The one divergence** (question `0977f2af`, knowledge-update, 2 gold sessions): both runs retrieved the identical top-10 session set; one gold session sits at rank 5 (published) vs rank 6 (re-run), moving that question's recall_all@5 from 1.0 to 0.5. The harness persists rankings rather than raw scores, so the exact score gap cannot be shown, but the identical top-10 set identifies this as cross-architecture floating-point noise on an RRF near-tie, not a retrieval failure. R@10 = 1.0 in both runs. + +**Reproduce:** + +```bash +python3 run_eval.py --data data/subset_kupd.json --output results_rerun --strategy default --resume +``` + +Raw artifacts are kept on the benchmark Modal volume (`uteke-longmemeval`, `default/` and rerun prefixes), consistent with the published run — datasets and result JSONL files are not committed to git (see `.gitignore` here; `download_data.sh` fetches the dataset). + +--- + ## Uteke Retrieval — Strategy Comparison ### Vector (semantic only) @@ -115,3 +137,45 @@ Binary built in-image from exact SHA bfbc296 (PR #1137/#1138), image build print Context: v0.15.0 hybrid baseline on the same dataset: Overall R@5 = 0.854 / R@10 = 0.885 (2026-08-13). Fusion default lifts full-500Q recall@5 by **+9.2 points** (0.854 → 0.946) with zero configuration. + +--- + +## Contradiction-resolution segment (#1172 Fase 3) — 2026-09-06 + +**Active-store knowledge-update segment**: 40 topics × (stale fact + winner fact + 3 distractors), +queries ask "which {thing} does {topic} use now?" (semantic, no keyword echo of the answer). +Baseline ranks with BOTH facts active; resolved ranks after `supersede(stale → winner)` — +baseline is measured for every strategy BEFORE any resolution, then the store is resolved once. +Binary: local release build (0.16.0 + #1185 ledger), local ONNX EmbeddingGemma, ARM64. + +Harness: `contradiction_segment.py` (this directory). Raw metrics: `results_contradiction_f3/metrics.json`. + +| Strategy | Stage | winner@1 | winner@5 | winner MRR | stale@1 | stale@5 | +|---|---|---|---|---|---|---| +| fusion (default) | baseline (unresolved) | 0.850 | 1.000 | 0.925 | 0.150 | **1.000** | +| fusion (default) | resolved (superseded) | **1.000** | 1.000 | **1.000** | 0.000 | 0.000 | +| hybrid | baseline | 0.025 | 1.000 | 0.469 | 0.975 | 1.000 | +| hybrid | resolved | 0.225 | 1.000 | 0.588 | 0.000 | 0.000 | +| vector | baseline | 0.950 | 1.000 | 0.975 | 0.050 | 1.000 | +| vector | resolved | 1.000 | 1.000 | 1.000 | 0.000 | 0.000 | + +Findings: + +- **Unresolved conflicts pollute every strategy's top-5**: with both facts active, the stale + fact sat in top-5 for 100% of topics on all strategies (hybrid's BM25 even ranks the stale + fact top-1 for 97.5% of topics — the old fact's "uses X" phrasing matches "use now?" + queries lexically). After `supersede`, stale@1 and stale@5 drop to **0.000** everywhere + (deprecated memories are excluded from recall). +- **Supersede lifts the default surface**: fusion winner@1 0.850 → 1.000, MRR 0.925 → 1.000; + vector 0.950 → 1.000. Hybrid stays weakest on winner@1 (lexical BM25 keeps the new fact's + "switched to" phrasing behind distractors) but its stale pollution is fully cleared. +- **Ledger integrity**: `contradictions list` listed all 40 resolutions; every stale fact is + restorable via `contradictions undo` (auditable conflict resolution, #1172 F2). + +Interpretation: ranking alone often picks the winner, but only explicit conflict resolution +guarantees stale facts leave the retrieval surface — the difference between "usually right" +(85–95% top-1) and deterministic freshness (100% top-1, zero stale). For agent memory, where +"use now" queries are the norm, resolution is what keeps top-1 trustworthy. This segment is +synthetic and deterministic (fixed topic list); it measures the conflict-resolution pipeline, +not LongMemEval dataset recall. + diff --git a/benchmarks/longmemeval/contradiction_segment.py b/benchmarks/longmemeval/contradiction_segment.py new file mode 100644 index 00000000..b6f205dd --- /dev/null +++ b/benchmarks/longmemeval/contradiction_segment.py @@ -0,0 +1,271 @@ +#!/usr/bin/env python3 +""" +Contradiction-segment benchmark (#1172 Fase 3). + +Measures how conflict resolution (supersede, #1053/#1172) affects retrieval +quality on a synthetic knowledge-update workload. This is the ACTIVE-store +counterpart to LongMemEval's passive knowledge-update subset (subset_kupd): +instead of asking whether stale sessions are retrieved, we resolve conflicts +in the store first and ask whether recall surfaces the WINNER. + +Design (deterministic, N topics): + 1. Seed 2 memories per topic: + - stale: the OLD fact ("… uses tool X") + - winner: the NEW fact ("… switched to tool Y") + plus D distractor memories per topic (same domain vocabulary, no conflict). + 2. Baseline run: both facts active (no supersede). Query each topic + semantically ("what does topic use now?"). Measures how often the + stale fact pollutes top-k when nothing resolved the conflict. + 3. Resolved run: supersede(stale → winner) via the CLI, then re-query. + Measures winner@k and stale@k on the RESOLVED store. + 4. Ledger sanity: contradiction_resolutions lists the pair; undo restores + (audited, then re-superseded). + +Metrics per strategy: + winner@{1,3,5} — winner memory ranked in top-k + stale@{1,5} — stale memory present in top-k (0.0 expected post-resolve) + MRR (winner) — reciprocal rank of the winner + +Usage: + python3 contradiction_segment.py --binary ../../target/release/uteke --topics 40 + python3 contradiction_segment.py --topics 40 --json out.json +""" + +import argparse +import json +import shutil +import subprocess +import sys +import tempfile +from pathlib import Path + +TOPICS = [ + ("acme-corp", "build tooling", "gradle", "bazel"), + ("brightpath", "package manager", "yarn", "pnpm"), + ("cloudline", "hosting provider", "heroku", "fly.io"), + ("dataworks", "message queue", "rabbitmq", "kafka"), + ("everhost", "web server", "apache", "nginx"), + ("fintrak", "ledger database", "postgresql", "cockroachdb"), + ("gadgethub", "mobile framework", "cordova", "flutter"), + ("heliosoft", "ci system", "jenkins", "github actions"), + ("innodata", "search engine", "elasticsearch", "meilisearch"), + ("jetstream", "api style", "soap", "grpc"), + ("kobalt", "css framework", "bootstrap", "tailwind"), + ("lumenpath", "state management", "redux", "zustand"), + ("metriq", "observability stack", "graphite", "prometheus"), + ("novabyte", "language runtime", "java", "kotlin"), + ("orbita", "container runtime", "docker swarm", "kubernetes"), + ("pixelbay", "image format", "jpeg-xl", "avif"), + ("quantex", "config format", "ini", "toml"), + ("riverbend", "version control", "svn", "git"), + ("saltmarsh", "auth protocol", "basic auth", "oauth2"), + ("tidewater", "template engine", "ejs", "handlebars"), + ("umbracloud", "storage layer", "mongodb", "sqlite"), + ("vellum", "docs generator", "javadoc", "rustdoc"), + ("wharfside", "package registry", "nexus", "ghcr"), + ("xenolith", "testing framework", "junit4", "junit5"), + ("yarrow", "scheduler", "cron", "systemd timers"), + ("zephyr", "linting tool", "tslint", "eslint"), + ("argonhold", "secret store", "env files", "vault"), + ("basaltix", "logging library", "log4j", "tracing"), + ("cobaltrun", "runtime monitor", "new relic", "otel"), + ("duskfield", "error tracker", "rollbar", "sentry"), + ("emberfall", "feature flags", "launchdarkly", "unleash"), + ("frostline", "cache layer", "memcached", "redis"), + ("glacierpeak", "object store", "s3 class", "r2"), + ("hollowpine", "markdown parser", "marked", "comrak"), + ("irisvale", "date library", "moment", "dayjs"), + ("jaderock", "http client", "axios", "fetch"), + ("kelpforest", "orm", "sequelize", "drizzle"), + ("lavaglass", "bundler", "webpack", "vite"), + ("mistvale", "type checker", "flow", "typescript"), + ("nightsky", "charting library", "chart.js", "d3"), +] + +# Question phrasings deliberately avoid the exact "uses/switched to" verbs +# so the query is semantic, not keyword lookup. +QUESTION = "which {thing} does {topic} use now?" + + +def resolve_binary(cli_path: str) -> str: + if cli_path: + p = Path(cli_path) + if p.exists(): + return str(p) + print(f"warning: --binary {cli_path} missing; falling back", file=sys.stderr) + repo = Path(__file__).resolve().parent.parent.parent + cand = repo / "target" / "release" / "uteke" + if cand.exists(): + return str(cand) + for c in ("/opt/data/.local/bin/uteke",): + if Path(c).exists(): + return str(c) + return shutil.which("uteke") or "uteke" + + +def uteke(binary: str, store: Path, namespace: str, args: list[str]) -> dict | list: + cmd = [ + binary, + "--store", str(store), + "--namespace", namespace, + "--json", + *args, + ] + result = subprocess.run(cmd, capture_output=True, text=True, timeout=600) + if result.returncode != 0: + raise RuntimeError(f"uteke {' '.join(args)} failed: {result.stderr[:400]}") + return json.loads(result.stdout) + + +def remember(binary, store, ns, content: str) -> str: + out = uteke(binary, store, ns, ["remember", content]) + return str(out["id"]) # type: ignore[no-any-return] + + +def supersede(binary, store, ns, old: str, new: str) -> None: + uteke(binary, store, ns, ["supersede", old, new, "--reason", "benchmark conflict resolution"]) + + +def recall_ids(binary, store, ns, query: str, strategy: str, k: int) -> list[str]: + out = uteke(binary, store, ns, [ + "recall", query, + "--limit", str(k), + "--min", "0.0", + "--strategy", strategy, + ]) + return [m["memory_id"] for m in out] + + +def recall_map(binary, store, ns) -> dict[str, str]: + """id → content for the namespace (to map ids back to roles).""" + out = uteke(binary, store, ns, ["list", "--limit", "500"]) + return {m["id"]: m["content"] for m in out} + + +def metrics_at(ranking: list[str], winner: str, stale: str) -> tuple[float, float, float, float, float]: + w = lambda k: 1.0 if winner in ranking[:k] else 0.0 + s = lambda k: 1.0 if stale in ranking[:k] else 0.0 + rr = 0.0 + for i, mid in enumerate(ranking, start=1): + if mid == winner: + rr = 1.0 / i + break + return w(1), w(5), rr, s(1), s(5) + + +def main() -> int: + ap = argparse.ArgumentParser(description="#1172 F3 contradiction segment") + ap.add_argument("--binary", default="", help="uteke binary path") + ap.add_argument("--store", default="", help="existing store to reuse (default: temp)") + ap.add_argument("--namespace", default="bench-contradiction") + ap.add_argument("--distractors", type=int, default=3, help="distractors per topic") + ap.add_argument("--topics", type=int, default=0, help="limit topics (0 = all)") + ap.add_argument("--strategy", default="fusion") + ap.add_argument("--json", default="", help="write metrics JSON here") + args = ap.parse_args() + + topics = TOPICS[: args.topics] if args.topics > 0 else TOPICS + binary = resolve_binary(args.binary) + print(f"binary: {binary}") + + tmp = None + if args.store: + store = Path(args.store) + else: + tmp = tempfile.TemporaryDirectory(prefix="uteke-contradiction-") + store = Path(tmp.name) / "bench.uteke" + + ns = args.namespace + roles: dict[str, tuple[str, str]] = {} # winner id → (stale id, topic) + topic_thing = {t: th for t, th, _o, _n in topics} + + # ── Seed ──────────────────────────────────────────────────────────── + print(f"seeding {len(topics)} topics (+{args.distractors} distractors each)…") + distractor_pool = [ + "weekly sync notes and standup summaries", + "onboarding checklist for new engineers", + "retro action items from the last sprint", + "vendor invoice and billing contacts", + "conference talk notes and takeaways", + ] + for topic, thing, old_tool, new_tool in topics: + stale = remember(binary, store, ns, + f"{topic} uses {old_tool} for {thing}. Decision recorded after evaluation.") + winner = remember(binary, store, ns, + f"{topic} switched to {new_tool} for {thing}. The old {old_tool} setup is retired.") + roles[winner] = (stale, topic) + for d in range(args.distractors): + remember(binary, store, ns, + f"{topic} {distractor_pool[d % len(distractor_pool)]} ({thing} context {d})") + + strategies = [s.strip() for s in args.strategy.split(",") if s.strip()] + results: dict[str, dict] = {} + + def measure(strategy: str) -> dict: + w1s = w5s = rrs = ss1 = ss5 = 0.0 + n = 0 + for winner, (stale, topic) in roles.items(): + q = QUESTION.format(thing=topic_thing[topic], topic=topic) + ranking = recall_ids(binary, store, ns, q, strategy, 10) + w1, w5, rr, s1, s5 = metrics_at(ranking, winner, stale) + w1s += w1; w5s += w5; rrs += rr; ss1 += s1; ss5 += s5 + n += 1 + return { + "winner@1": w1s / n, "winner@5": w5s / n, "winner_mrr": rrs / n, + "stale@1": ss1 / n, "stale@5": ss5 / n, "n": n, + } + + # ── Baseline for ALL strategies on the UNRESOLVED store first ────── + # (code-scanning fix: resolving per-strategy left later strategies + # measuring "baseline" on an already-resolved store.) + baselines = {s: measure(s) for s in strategies} + + # ── Resolve once: supersede every stale → winner ─────────────────── + for winner, (stale, _topic) in roles.items(): + supersede(binary, store, ns, stale, winner) + + # Ledger sanity: every resolution is listed (F2 surface, in-loop). + ledger_raw = subprocess.run( + [binary, "--store", str(store), "--json", + "contradictions", "list", "--namespace", ns, "--limit", "500"], + capture_output=True, text=True, timeout=600, + ) + ledger = json.loads(ledger_raw.stdout) + listed = {e["id"] for e in ledger} + ledger_ok = all(stale in listed for _w, (stale, _t) in roles.items()) + + # ── Resolved metrics for all strategies ──────────────────────────── + resolved = {s: measure(s) for s in strategies} + + for strategy in strategies: + baseline = {k: v for k, v in baselines[strategy].items() if k != "n"} + res = {k: v for k, v in resolved[strategy].items() if k != "n"} + n = baselines[strategy]["n"] + results[strategy] = { + "questions": n, + "baseline_unresolved": baseline, + "resolved": res, + "ledger_listed_all": ledger_ok, + } + print(f"\n[{strategy}] n={n}") + print(f" baseline (unresolved): winner@1={baseline['winner@1']:.3f} " + f"winner@5={baseline['winner@5']:.3f} MRR={baseline['winner_mrr']:.3f} " + f"stale@1={baseline['stale@1']:.3f} stale@5={baseline['stale@5']:.3f}") + print(f" resolved (superseded): winner@1={res['winner@1']:.3f} " + f"winner@5={res['winner@5']:.3f} MRR={res['winner_mrr']:.3f} " + f"stale@1={res['stale@1']:.3f} stale@5={res['stale@5']:.3f}") + print(f" ledger lists all resolutions: {ledger_ok}") + + if args.json: + out = Path(args.json) + out.parent.mkdir(parents=True, exist_ok=True) + out.write_text(json.dumps(results, indent=2)) + print(f"\nmetrics → {out}") + + if tmp: + tmp.cleanup() + return 0 + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/crates/uteke-cli/Cargo.toml b/crates/uteke-cli/Cargo.toml index c95cab3a..a668a152 100644 --- a/crates/uteke-cli/Cargo.toml +++ b/crates/uteke-cli/Cargo.toml @@ -14,14 +14,15 @@ name = "uteke" path = "src/main.rs" [features] -default = ["usearch"] -# Vector index backend forwarded to uteke-core. Exactly one must be enabled; -# default is usearch (HNSW C++ FFI). vecq = pure-Rust 4-bit quantization (#1098). +# Dual-engine default (#1168): ship BOTH engines, runtime-selectable via +# UTEKE_VECTOR_BACKEND / [vector] backend. Slim builds: --no-default-features +# --features vecq (mobile, pure Rust) or --features usearch (classic). +default = ["usearch", "vecq"] usearch = ["uteke-core/usearch"] vecq = ["uteke-core/vecq"] [dependencies] -uteke-core = { path = "../uteke-core", version = "0.16.0", default-features = false, features = ["onnx"] } +uteke-core = { path = "../uteke-core", version = "0.17.0", default-features = false, features = ["onnx"] } clap = { version = "4", features = ["derive"] } serde_json = "1" tracing = "0.1" diff --git a/crates/uteke-cli/src/cli.rs b/crates/uteke-cli/src/cli.rs index ef1db9a3..f93dc887 100644 --- a/crates/uteke-cli/src/cli.rs +++ b/crates/uteke-cli/src/cli.rs @@ -124,6 +124,9 @@ pub enum Commands { /// When absent, recency uses the default weight (0.1). Use --no-recency to disable (#721). #[arg(long)] recency: Option, + /// Explain each result: show the ranking signals behind it (#1160) + #[arg(long)] + explain: bool, /// Follow relationship edges in memory metadata #[arg(long)] related: bool, @@ -460,6 +463,26 @@ pub enum Commands { #[arg(long, default_value = "20")] limit: usize, }, + /// Show the full provenance report for a memory (#1172) + Provenance { + /// Memory ID (UUID) + id: String, + }, + /// Resolve a conflict: mark old_id superseded by new_id (#1053) + Supersede { + /// Full UUID or unambiguous prefix of the STALE memory + old: String, + /// Full UUID or unambiguous prefix of the CURRENT memory + new: String, + /// Why it was superseded (stored on the deprecation) + #[arg(long)] + reason: Option, + }, + /// Inspect the contradiction resolution ledger (#1172) + Contradictions { + #[command(subcommand)] + command: ContradictionCommands, + }, /// Document operations — wiki/knowledge base (#406, #411) Doc { #[command(subcommand)] @@ -666,6 +689,25 @@ pub enum TagCommands { }, } +/// Subcommands for the contradiction resolution ledger (#1172). +#[derive(Subcommand)] +pub enum ContradictionCommands { + /// List superseded-but-not-restored memories (the resolution ledger) + List { + /// Filter by namespace + #[arg(long)] + namespace: Option, + /// Maximum entries to show + #[arg(long, default_value = "50")] + limit: usize, + }, + /// Restore a superseded memory — undoes the supersession pair + Undo { + /// Memory ID (UUID) of the retired memory to restore + id: String, + }, +} + /// Subcommands for namespace management. #[derive(Subcommand)] pub enum NamespaceCommands { @@ -681,6 +723,34 @@ pub enum NamespaceCommands { /// Namespace name to set as default name: String, }, + /// Move a memory to another namespace (#1181) + Move { + /// Memory ID + id: String, + /// Target namespace + namespace: String, + }, + /// Rename a namespace — merges into the target when it already exists (#1181) + Rename { + /// Current namespace name + from: String, + /// New namespace name + to: String, + }, + /// Delete a namespace with an explicit strategy for its memories (#1181) + Delete { + /// Namespace to delete + name: String, + /// What happens to its memories: refuse (default), merge, deprecate + #[arg(long, default_value = "refuse")] + strategy: String, + /// Target namespace when strategy = merge + #[arg(long)] + target: Option, + /// Confirm the deletion (required) + #[arg(long)] + confirm: bool, + }, } /// Feedback actions for trust scoring (#718). diff --git a/crates/uteke-cli/src/commands/contradictions.rs b/crates/uteke-cli/src/commands/contradictions.rs new file mode 100644 index 00000000..70057b8b --- /dev/null +++ b/crates/uteke-cli/src/commands/contradictions.rs @@ -0,0 +1,130 @@ +//! Contradiction ledger subcommands — list, undo (#1172 Fase 2). + +use crate::cli::Cli; +use crate::cli::ContradictionCommands; +use crate::output; +use uteke_core::Uteke; + +pub(crate) fn run(cli: &Cli, uteke: &Uteke, command: &ContradictionCommands) -> Result<(), String> { + match command { + ContradictionCommands::List { namespace, limit } => { + tracing::info!( + "Listing contradiction resolutions (namespace={namespace:?}, limit={limit})" + ); + let resolutions = uteke + .contradiction_resolutions(namespace.as_deref(), *limit) + .map_err(|e| format!("Failed to list contradiction resolutions: {e}"))?; + if cli.json { + output::print_json(&resolutions); + } else if resolutions.is_empty() { + println!("No contradiction resolutions found."); + } else { + println!("Contradiction resolutions ({} total):\n", resolutions.len()); + for r in &resolutions { + let deprecated_at = r + .deprecated_at + .as_ref() + .map(|d| d.format("%Y-%m-%d %H:%M").to_string()) + .unwrap_or_else(|| "unknown".into()); + println!(" {} [{}]", short_id(&r.id), deprecated_at); + let preview: String = r.content.chars().take(60).collect(); + println!(" {}", preview); + println!( + " reason: {}", + r.deprecate_reason.as_deref().unwrap_or("—") + ); + } + } + Ok(()) + } + ContradictionCommands::Undo { id } => { + tracing::info!("Undoing supersession for memory {id}"); + // Accept full UUID or unambiguous prefix (same contract as MCP). + let resolved = if id.len() == 36 { + id.clone() + } else { + match uteke + .resolve_id_prefix(id) + .map_err(|e| format!("Failed to resolve id: {e}"))? + { + Some(full) => full, + None => return Err(format!("No memory matches id prefix '{id}'")), + } + }; + match uteke + .undo_supersession(&resolved) + .map_err(|e| format!("Failed to undo supersession: {e}"))? + { + Some(winner) => { + if cli.json { + output::print_json(&serde_json::json!({ + "undone": true, + "restored": resolved, + "was_superseded_by": winner, + })); + } else { + println!("✓ Restored memory {resolved}"); + println!(" was superseded by {winner}"); + println!(" supersession edges removed — the pair is no longer flagged"); + } + } + None => { + return Err(format!("No supersession found for memory: {id}")); + } + } + Ok(()) + } + } +} + +/// Short ID helper for the human-readable ledger listing. +fn short_id(id: &str) -> String { + id.chars().take(8).collect() +} + +/// `uteke supersede old new [--reason text]` (#1053) — CLI parity with the +/// MCP tool and HTTP surface. Accepts full UUIDs or unambiguous prefixes. +pub(crate) fn supersede( + cli: &Cli, + uteke: &Uteke, + old: &str, + new: &str, + reason: Option<&str>, +) -> Result<(), String> { + tracing::info!("Superseding {old} -> {new}"); + let resolve = |id: &str| -> Result { + if id.len() == 36 { + return Ok(id.to_string()); + } + match uteke + .resolve_id_prefix(id) + .map_err(|e| format!("Failed to resolve id: {e}"))? + { + Some(full) => Ok(full), + None => Err(format!("No memory matches id prefix '{id}'")), + } + }; + let old_id = resolve(old)?; + let new_id = resolve(new)?; + + let (o, n) = uteke + .supersede(&old_id, &new_id, reason) + .map_err(|e| format!("Failed to supersede: {e}"))?; + if cli.json { + output::print_json(&serde_json::json!({ + "superseded": o, + "by": n, + "reason": reason, + })); + } else { + println!("✓ Superseded {} → {}", short_id(&o), short_id(&n)); + if let Some(r) = reason { + println!(" reason: {r}"); + } + println!( + " recall now flags the pair; restore: uteke contradictions undo {}", + short_id(&o) + ); + } + Ok(()) +} diff --git a/crates/uteke-cli/src/commands/maintenance.rs b/crates/uteke-cli/src/commands/maintenance.rs index 26e955d9..ec2d9016 100644 --- a/crates/uteke-cli/src/commands/maintenance.rs +++ b/crates/uteke-cli/src/commands/maintenance.rs @@ -63,19 +63,37 @@ pub(crate) fn run_repair( tracing::info!("Running repair"); } - let report = uteke.repair().map_err(|e| format!("Repair failed: {e}"))?; + let mut report = uteke.repair().map_err(|e| format!("Repair failed: {e}"))?; + + // Optional: re-embed memories with missing vectors. + // Runs BEFORE the Repair Report is printed so `index_after` and the + // "still differs" warning reflect the final state, not the intermediate + // rebuild that legitimately excludes NULL/empty-embedding rows (#1149). + let reembed_report = if reembed { + tracing::info!("Running repair --reembed (regenerating missing embeddings)"); + match uteke.reembed_missing() { + Ok(r) => { + // Refresh the vector count to include newly appended vectors. + if let Ok(v) = uteke.verify() { + report.index_after = v.index_count; + } + Some(Ok(r)) + } + Err(e) => Some(Err(e)), + } + } else { + None + }; + + // Repair itself succeeded — always print its report (even if reembed failed). if cli.json { output::print_json(&report); } else { output::print_repair_human(&report); } - // Optional: re-embed memories with missing vectors. - if reembed { - tracing::info!("Running repair --reembed (regenerating missing embeddings)"); - let reembed_report = uteke - .reembed_missing() - .map_err(|e| format!("Re-embed failed: {e}"))?; + if let Some(result) = reembed_report { + let reembed_report = result.map_err(|e| format!("Re-embed failed: {e}"))?; if cli.json { output::print_json(&reembed_report); } else { diff --git a/crates/uteke-cli/src/commands/mod.rs b/crates/uteke-cli/src/commands/mod.rs index 3ccdc902..83943dc7 100644 --- a/crates/uteke-cli/src/commands/mod.rs +++ b/crates/uteke-cli/src/commands/mod.rs @@ -2,6 +2,7 @@ mod aging; pub(crate) mod bench; +mod contradictions; mod doc; mod dream; mod edges; @@ -87,6 +88,7 @@ pub(crate) fn run_command(cli: &Cli, uteke: &mut Uteke, config: &Config) -> Resu r#where, r#type, enrich, + explain, } => recall::run_recall( cli, uteke, @@ -110,6 +112,7 @@ pub(crate) fn run_command(cli: &Cli, uteke: &mut Uteke, config: &Config) -> Resu *recency, r#type.as_deref(), *enrich, + *explain, ), Commands::Context { namespace } => { @@ -384,6 +387,67 @@ pub(crate) fn run_command(cli: &Cli, uteke: &mut Uteke, config: &Config) -> Resu Commands::Timeline { id, limit } => timeline::run(cli, uteke, id, *limit), + Commands::Contradictions { command } => { + crate::commands::contradictions::run(cli, uteke, command) + } + + Commands::Supersede { old, new, reason } => { + crate::commands::contradictions::supersede(cli, uteke, old, new, reason.as_deref()) + } + + Commands::Provenance { id } => { + let report = uteke + .provenance(id) + .map_err(|e| format!("Failed to read provenance: {e}"))?; + match report { + Some(report) if cli.json => crate::output::print_json(&report), + Some(report) => { + println!("Provenance for memory {id}"); + println!(" Namespace: {}", report.namespace); + println!(" Author type: {}", report.author_type); + println!( + " Source: {} ({})", + report.source.as_deref().unwrap_or("—"), + report.source_type + ); + println!(" Trust tier: {:?}", report.trust_tier); + println!(" Created: {}", report.created_at); + println!(" Updated: {}", report.updated_at); + println!(" Deprecated: {}", report.deprecated); + match report.source_hash.as_deref() { + Some(h) if h == report.content_hash_now => { + println!(" Content hash: {h} ✓ (matches write-time hash)"); + } + Some(h) => { + println!( + " Content hash: {h} ✗ MISMATCH — content changed after write (now {})", + report.content_hash_now + ); + } + None => { + println!( + " Content hash: — (pre-v18 row; now {})", + report.content_hash_now + ); + } + } + println!("\n Event chain ({} events):", report.events.len()); + for event in &report.events { + let actor = event.actor.as_deref().unwrap_or("—"); + println!( + " • [{}] {} (actor: {actor})", + event.created_at, event.event_type + ); + if let Some(evidence) = &event.evidence { + println!(" evidence: {evidence}"); + } + } + } + None => return Err(format!("Memory not found: {id}")), + } + Ok(()) + } + Commands::Doc { command } => crate::commands::doc::run(cli, uteke, command, config), Commands::Upgrade { yes } => upgrade::run(*yes), diff --git a/crates/uteke-cli/src/commands/namespace.rs b/crates/uteke-cli/src/commands/namespace.rs index 40d92277..2f4a3c93 100644 --- a/crates/uteke-cli/src/commands/namespace.rs +++ b/crates/uteke-cli/src/commands/namespace.rs @@ -50,5 +50,79 @@ pub(crate) fn run(cli: &Cli, uteke: &Uteke, command: &NamespaceCommands) -> Resu } Ok(()) } + NamespaceCommands::Move { id, namespace } => { + tracing::info!("Moving memory {id} to namespace '{namespace}'"); + let moved = uteke + .move_memory(id, namespace) + .map_err(|e| format!("Failed to move memory: {e}"))?; + if !moved { + return Err(format!("Memory not found: {id}")); + } + if cli.json { + output::print_json(&serde_json::json!({ + "moved": true, + "id": id, + "namespace": namespace, + })); + } else { + println!("\u{2713} Moved memory {id} to namespace '{namespace}'"); + } + Ok(()) + } + NamespaceCommands::Rename { from, to } => { + tracing::info!("Renaming namespace '{from}' to '{to}'"); + let result = uteke + .rename_namespace(from, to) + .map_err(|e| format!("Failed to rename namespace: {e}"))?; + if cli.json { + output::print_json(&result); + } else { + let kind = if result.target_existed { + "merged into existing" + } else { + "renamed to" + }; + println!( + "\u{2713} Namespace '{from}' {kind} '{to}' — {} memories moved", + result.moved + ); + } + Ok(()) + } + NamespaceCommands::Delete { + name, + strategy, + target, + confirm, + } => { + if !confirm { + return Err( + "Refusing to delete a namespace without --confirm (this affects its memories)" + .to_string(), + ); + } + tracing::info!("Deleting namespace '{name}' (strategy={strategy})"); + let result = uteke + .delete_namespace(name, strategy, target.as_deref()) + .map_err(|e| format!("Failed to delete namespace: {e}"))?; + if cli.json { + output::print_json(&result); + } else { + match result.strategy.as_str() { + "merge" => println!( + "\u{2713} Moved {} memories from '{}' to '{}' — namespace removed", + result.affected, + result.name, + result.target.as_deref().unwrap_or("?") + ), + "deprecate" => println!( + "\u{2713} Soft-deleted {} memories in '{}' (restorable via promote; the name stays visible as deprecated-only)", + result.affected, result.name + ), + _ => println!("\u{2713} Namespace '{}' deleted", result.name), + } + } + Ok(()) + } } } diff --git a/crates/uteke-cli/src/commands/recall.rs b/crates/uteke-cli/src/commands/recall.rs index 1d73a2e2..f91093ee 100644 --- a/crates/uteke-cli/src/commands/recall.rs +++ b/crates/uteke-cli/src/commands/recall.rs @@ -30,6 +30,7 @@ pub(crate) fn run_recall( recency: Option, search_type: Option<&str>, enrich: bool, + explain: bool, ) -> Result<(), String> { // Resolve search type: --type flag > default (All = unified) let resolved_search_type = match search_type { @@ -108,6 +109,99 @@ pub(crate) fn run_recall( }, }); + // --explain (#1160): memory-recall explanation mode. Runs the real + // strategy pipeline with per-stage instrumentation and prints the + // signals behind each result. Mutually exclusive with the unified + // search surface (docs/entities) — explanation is memory-only. An + // omitted --type (the default invocation) is accepted and treated as + // memory recall; only explicit --type doc is rejected (code-scanning + // fix: None maps to SearchType::All, which the old guard rejected). + if explain { + if resolved_search_type == SearchType::Document { + return Err( + "--explain works on memory recall; --type doc is not supported.".to_string(), + ); + } + if at.is_some() { + return Err("--explain and --at cannot be used together".to_string()); + } + if related { + return Err("--explain and --related cannot be used together".to_string()); + } + if entity.is_some() { + return Err("--explain and --entity cannot be used together".to_string()); + } + if category.is_some() { + return Err("--explain and --category cannot be used together".to_string()); + } + if where_filter.is_some() { + return Err("--explain and --where cannot be used together".to_string()); + } + if context { + return Err("--explain and --context cannot be used together".to_string()); + } + if enrich { + return Err("--explain and --enrich cannot be used together".to_string()); + } + let explained = uteke + .recall_explained(query, limit, tags_filter, ns, resolved_strategy, min_score) + .map_err(|e| format!("Failed to recall: {e}"))?; + uteke.reset_salience_recency_config(); + + if explained.is_empty() { + if cli.json { + output::print_json(&explained); + } else { + println!("No matching memories found."); + } + return Ok(()); + } + + if cli.json { + output::print_json(&explained); + } else { + for (i, er) in explained.iter().enumerate() { + let ex = &er.explanation; + println!("{}. {}", i + 1, er.result.memory.content); + println!(" Final score: {:.4} ({})", ex.final_score, ex.strategy); + if let Some(sim) = ex.vector_similarity { + println!(" Vector similarity: {:.4}", sim); + } + if let Some(rank) = ex.vector_rank { + println!(" Vector rank: #{}", rank); + } + if let Some(rank) = ex.fts_rank { + println!(" FTS rank: #{}", rank); + } + if let Some(rrf) = ex.rrf_score { + println!(" RRF score: {:.4}", rrf); + } + if let Some(c) = ex.fusion_vector_contribution { + println!(" Fusion vector contrib: {:.4}", c); + } + if let Some(c) = ex.fusion_hybrid_contribution { + println!(" Fusion hybrid contrib: {:.4}", c); + } + if let Some(b) = ex.jaccard_boost { + println!(" Jaccard boost: +{:.4}", b); + } + if let Some(b) = ex.salience_boost { + println!(" Salience boost: +{:.4}", b); + } + if let Some(b) = ex.recency_boost { + println!(" Recency boost: +{:.4}", b); + } + if let Some(b) = ex.graph_boost { + println!(" Graph boost: +{:.4}", b); + } + println!(" Base score: {:.4}", ex.base_score); + println!(" ID: {}", er.result.memory.id); + println!(); + } + } + return Ok(()); + } + if use_unified { // Unified search path (#531) let unified_results = uteke diff --git a/crates/uteke-cli/src/commands/room.rs b/crates/uteke-cli/src/commands/room.rs index 16d3cec2..30bc456d 100644 --- a/crates/uteke-cli/src/commands/room.rs +++ b/crates/uteke-cli/src/commands/room.rs @@ -160,14 +160,22 @@ pub(crate) fn run( return Err("Operation not confirmed".to_string()); } - uteke + let unlinked = uteke .delete_room(room_id) .map_err(|e| format!("Failed to delete room: {e}"))?; if cli.json { - println!("{}", serde_json::json!({"deleted": room_id})); + println!( + "{}", + serde_json::json!({ + "deleted": room_id, + "unlinked_memories": unlinked, + }) + ); } else { - println!("Room {room_id} deleted. Memories are preserved in their namespaces."); + println!( + "Room {room_id} deleted. {unlinked} memory link(s) removed; memories are preserved in their namespaces." + ); } Ok(()) } diff --git a/crates/uteke-cli/src/config.rs b/crates/uteke-cli/src/config.rs index 2c92eaac..2c19ac77 100644 --- a/crates/uteke-cli/src/config.rs +++ b/crates/uteke-cli/src/config.rs @@ -26,6 +26,16 @@ impl Default for StoreConfig { } } +/// Vector engine configuration (#1168). +/// +/// Which engine the index runs on when BOTH are compiled in. Slim builds +/// (one engine) ignore this — the compiled-in engine always runs. +#[derive(serde::Deserialize, Clone, Default)] +pub struct VectorConfig { + /// "usearch" or "vecq". Empty = compiled-in default (usearch when present). + pub backend: String, +} + /// Embedding model configuration. #[derive(serde::Deserialize, Clone)] #[serde(default)] @@ -430,6 +440,8 @@ impl LifecycleConfig { #[serde(default)] pub struct Config { pub store: StoreConfig, + #[serde(default)] + pub vector: VectorConfig, pub embedding: EmbeddingConfig, pub extraction: ExtractionConfig, pub embed_fallback: EmbedFallbackConfig, @@ -457,6 +469,7 @@ impl Default for Config { fn default() -> Self { Self { store: StoreConfig::default(), + vector: VectorConfig::default(), embedding: EmbeddingConfig::default(), extraction: ExtractionConfig::default(), embed_fallback: EmbedFallbackConfig::default(), @@ -878,6 +891,14 @@ impl Config { } } + // Vector engine override (#1168). Ignored when the requested engine + // is not compiled in (resolution falls back with a warning in core). + if let Ok(v) = std::env::var("UTEKE_VECTOR_BACKEND") { + if !v.is_empty() { + self.vector.backend = v; + } + } + // Embedding backend overrides (#337) if let Ok(v) = std::env::var("UTEKE_EMBEDDING_BACKEND") { if !v.is_empty() { @@ -1818,4 +1839,235 @@ max_seq_length = 128 assert_eq!(cfg.server.port, 8767); assert!((cfg.recall.min_score - 0.3).abs() < f64::EPSILON); } + + // ── #1078 P0 batch 3: score range guards, strategy validation, graph + // weights, embed fallback — none previously covered. ───────────────── + + #[test] + #[serial_test::serial] + fn env_override_min_score_out_of_range_ignored() { + unsafe { + std::env::set_var("UTEKE_RECALL_MIN_SCORE", "1.5"); + } + let cfg = Config::default().apply_env_overrides(); + unsafe { + std::env::remove_var("UTEKE_RECALL_MIN_SCORE"); + } + assert!((cfg.recall.min_score - 0.3).abs() < f64::EPSILON); + } + + #[test] + #[serial_test::serial] + fn env_override_min_score_not_a_number_ignored() { + unsafe { + std::env::set_var("UTEKE_RECALL_MIN_SCORE", "banana"); + } + let cfg = Config::default().apply_env_overrides(); + unsafe { + std::env::remove_var("UTEKE_RECALL_MIN_SCORE"); + } + assert!((cfg.recall.min_score - 0.3).abs() < f64::EPSILON); + } + + #[test] + #[serial_test::serial] + fn env_override_strategy_valid_values_accepted() { + for v in ["vector", "fts5", "hybrid", "graph", "fusion"] { + unsafe { + std::env::set_var("UTEKE_RECALL_STRATEGY", v); + } + let cfg = Config::default().apply_env_overrides(); + assert_eq!(cfg.recall.default_strategy, v, "strategy {v} rejected"); + } + unsafe { + std::env::remove_var("UTEKE_RECALL_STRATEGY"); + } + } + + #[test] + #[serial_test::serial] + fn env_override_strategy_invalid_ignored() { + unsafe { + std::env::set_var("UTEKE_RECALL_STRATEGY", "quantum"); + } + let cfg = Config::default().apply_env_overrides(); + unsafe { + std::env::remove_var("UTEKE_RECALL_STRATEGY"); + } + // Invalid strategy keeps the fusion default + assert_eq!(cfg.recall.default_strategy, "fusion"); + } + + #[test] + #[serial_test::serial] + fn env_override_graph_weights_valid() { + unsafe { + std::env::set_var("UTEKE_GRAPH_DENSITY_WEIGHT", "0.5"); + std::env::set_var("UTEKE_GRAPH_AUTHORITY_WEIGHT", "0.7"); + } + let cfg = Config::default().apply_env_overrides(); + unsafe { + std::env::remove_var("UTEKE_GRAPH_DENSITY_WEIGHT"); + std::env::remove_var("UTEKE_GRAPH_AUTHORITY_WEIGHT"); + } + assert!((cfg.recall.graph_density_weight - 0.5).abs() < f32::EPSILON); + assert!((cfg.recall.graph_authority_weight - 0.7).abs() < f32::EPSILON); + } + + #[test] + #[serial_test::serial] + fn env_override_graph_weights_out_of_range_ignored() { + unsafe { + std::env::set_var("UTEKE_GRAPH_DENSITY_WEIGHT", "-1.0"); + std::env::set_var("UTEKE_GRAPH_AUTHORITY_WEIGHT", "2.0"); + } + let cfg = Config::default().apply_env_overrides(); + unsafe { + std::env::remove_var("UTEKE_GRAPH_DENSITY_WEIGHT"); + std::env::remove_var("UTEKE_GRAPH_AUTHORITY_WEIGHT"); + } + assert!((cfg.recall.graph_density_weight - 0.1).abs() < f32::EPSILON); + assert!((cfg.recall.graph_authority_weight - 0.1).abs() < f32::EPSILON); + } + + #[test] + #[serial_test::serial] + fn env_override_embed_fallback_all_fields() { + unsafe { + std::env::set_var("UTEKE_EMBED_FALLBACK_API_KEY", "fb-key"); + std::env::set_var("UTEKE_EMBED_FALLBACK_BASE_URL", "https://fb.example"); + std::env::set_var("UTEKE_EMBED_FALLBACK_ENDPOINT_PATH", "/v1/e"); + std::env::set_var("UTEKE_EMBED_FALLBACK_MODEL", "fb-model"); + } + let cfg = Config::default().apply_env_overrides(); + unsafe { + std::env::remove_var("UTEKE_EMBED_FALLBACK_API_KEY"); + std::env::remove_var("UTEKE_EMBED_FALLBACK_BASE_URL"); + std::env::remove_var("UTEKE_EMBED_FALLBACK_ENDPOINT_PATH"); + std::env::remove_var("UTEKE_EMBED_FALLBACK_MODEL"); + } + assert_eq!(cfg.embed_fallback.api_key, "fb-key"); + assert_eq!(cfg.embed_fallback.base_url, "https://fb.example"); + assert_eq!(cfg.embed_fallback.endpoint_path, "/v1/e"); + assert_eq!(cfg.embed_fallback.model, "fb-model"); + assert!(cfg.embed_fallback.is_configured()); + } + + #[test] + fn embed_fallback_is_configured_matrix() { + // Nothing set — not configured + let none = EmbedFallbackConfig { + api_key: String::new(), + base_url: String::new(), + endpoint_path: String::new(), + model: String::new(), + }; + assert!(!none.is_configured()); + + // Partial (2 of 3 required) — still not configured + let partial = EmbedFallbackConfig { + api_key: "k".into(), + base_url: "u".into(), + endpoint_path: String::new(), + model: String::new(), + }; + assert!(!partial.is_configured()); + + // All three required set — configured (endpoint_path optional) + let full = EmbedFallbackConfig { + api_key: "k".into(), + base_url: "u".into(), + endpoint_path: String::new(), + model: "m".into(), + }; + assert!(full.is_configured()); + } + + // ── #1078 P0 batch 5: mutation survivors — migrate_content, + // global_config_path, set_namespace_in_toml. Previously untested. + + #[test] + fn migrate_content_moves_embedding_keys_to_section() { + let old = "store_path = ~/.uteke\nmodel = gemma\nmax_seq_length = 512\n"; + let out = migrate_content(old); + assert!( + out.contains("[store]\npath = ~/.uteke"), + "store path: {out}" + ); + assert!( + out.contains("[embedding]\nmodel = gemma\nmax_seq_length = 512"), + "embedding keys must move to [embedding]: {out}" + ); + } + + #[test] + fn migrate_content_passes_through_unknown_and_sections() { + let old = "# comment\nunknown_key = 1\n[new_section]\nfoo = bar\n"; + let out = migrate_content(old); + assert!(out.contains("# comment")); + assert!(out.contains("unknown_key = 1")); + assert!(out.contains("[new_section]\nfoo = bar")); + assert!(!out.contains("[store]"), "no store keys: {out}"); + assert!(!out.contains("[embedding]"), "no embedding keys: {out}"); + } + + #[test] + #[serial_test::serial] + fn global_config_path_respects_uteke_home() { + unsafe { + std::env::set_var("UTEKE_HOME", "/tmp/uteke-home-test"); + } + let p = global_config_path().expect("UTEKE_HOME set → Some"); + assert_eq!(p, PathBuf::from("/tmp/uteke-home-test/uteke.toml")); + unsafe { + std::env::remove_var("UTEKE_HOME"); + } + } + + #[test] + fn set_namespace_rewrites_only_store_section() { + // namespace in [other] must NOT be rewritten (guards the != and && + // mutants: section-boundary tracking). + let content = "[store]\nnamespace = \"old\"\npath = p\n[other]\nnamespace = \"keep\"\n"; + let out = set_namespace_in_toml(content, "new"); + assert!(out.contains("namespace = \"new\"")); + assert!( + out.contains("namespace = \"keep\""), + "[other] namespace untouched: {out}" + ); + assert!(out.contains("path = p"), "other keys preserved: {out}"); + } + + #[test] + fn set_namespace_inserts_immediately_after_store_header() { + // namespace must land right after [store], not before it (guards + // the pos+1 insert-position mutant). + let content = "[store]\npath = p\n"; + let out = set_namespace_in_toml(content, "ns1"); + let lines: Vec<&str> = out.lines().collect(); + let pos = lines.iter().position(|l| *l == "[store]").unwrap(); + assert_eq!( + lines[pos + 1], + "namespace = \"ns1\"", + "insert after header: {out}" + ); + assert!(out.contains("path = p"), "existing keys kept: {out}"); + } + + #[test] + fn set_namespace_appends_store_section_when_missing() { + let out = set_namespace_in_toml("top = 1\n", "ns2"); + assert!( + out.contains("[store]\nnamespace = \"ns2\""), + "appended [store]: {out}" + ); + assert!(out.contains("top = 1")); + } + + #[test] + fn set_namespace_replaces_existing_value_in_place() { + let out = set_namespace_in_toml("[store]\nnamespace = \"a\"\n", "b"); + assert!(out.contains("namespace = \"b\"")); + assert!(!out.contains("\"a\""), "old value gone: {out}"); + } } diff --git a/crates/uteke-cli/src/main.rs b/crates/uteke-cli/src/main.rs index a2862f7c..712c4076 100644 --- a/crates/uteke-cli/src/main.rs +++ b/crates/uteke-cli/src/main.rs @@ -174,6 +174,9 @@ fn main() { authority_weight: config.recall.graph_authority_weight, enabled: config.recall.graph_rerank_enabled, }, + // Runtime vector-engine preference from uteke.toml [vector] (#1168). + // UTEKE_VECTOR_BACKEND env (higher precedence) is read inside core. + Some(config.vector.backend.as_str()), ) { Ok(mut u) => { // #719: apply Jaccard weight from config @@ -282,4 +285,39 @@ mod resolve_store_tests { unsafe { std::env::remove_var("UTEKE_HOME") }; assert_eq!(resolve_store_path(&cli, &cfg), expanded_default); } + + /// Namespace precedence (#1078 P0 batch 2). All cases mutate the + /// process-global UTEKE_NAMESPACE, so they run sequentially inside ONE + /// #[test] — same rationale as store_resolution_precedence above. + #[test] + fn namespace_resolution_precedence() { + let mut cfg = Config::default(); + + // Baseline: nothing set → "default" + unsafe { std::env::remove_var("UTEKE_NAMESPACE") }; + let mut cli = Cli::try_parse_from(["uteke", "stats"]).unwrap(); + assert_eq!(resolve_namespace(&cli, &cfg), "default"); + + // 1) --namespace flag beats env AND config + cfg.store.namespace = "from-config".to_string(); + unsafe { std::env::set_var("UTEKE_NAMESPACE", "from-env") }; + cli.namespace = Some("from-flag".to_string()); + assert_eq!(resolve_namespace(&cli, &cfg), "from-flag"); + + // 2) UTEKE_NAMESPACE env beats config when flag absent + cli.namespace = None; + assert_eq!(resolve_namespace(&cli, &cfg), "from-env"); + + // 3) empty env is ignored, falls back to config + unsafe { std::env::set_var("UTEKE_NAMESPACE", "") }; + assert_eq!(resolve_namespace(&cli, &cfg), "from-config"); + + // 4) config namespace used when no flag, no env + unsafe { std::env::remove_var("UTEKE_NAMESPACE") }; + assert_eq!(resolve_namespace(&cli, &cfg), "from-config"); + + // 5) config "default" literal falls through to "default" + cfg.store.namespace = "default".to_string(); + assert_eq!(resolve_namespace(&cli, &cfg), "default"); + } } diff --git a/crates/uteke-core/Cargo.toml b/crates/uteke-core/Cargo.toml index c5075e3f..47018304 100644 --- a/crates/uteke-core/Cargo.toml +++ b/crates/uteke-core/Cargo.toml @@ -11,7 +11,11 @@ categories = ["science", "data-structures"] readme = "../../README.md" [features] -default = ["onnx", "usearch"] +# Dual-engine default (#1168): official binaries ship BOTH engines so +# UTEKE_VECTOR_BACKEND / [vector] backend can switch at runtime. +# Slim builds: --no-default-features --features "onnx,vecq" (mobile, no C++) +# or --features "onnx,usearch" (classic single-engine desktop). +default = ["onnx", "usearch", "vecq"] onnx = ["dep:ort", "dep:ndarray", "dep:tokenizers"] # Default vector search backend (HNSW, C++ FFI). usearch = ["dep:usearch"] @@ -26,7 +30,7 @@ serde = { version = "1", features = ["derive"] } serde_json = "1" rusqlite = { version = "0.40", features = ["bundled"] } usearch = { version = "2", optional = true } -vecq-core = { version = "0.1", optional = true } +vecq-core = { version = "0.3", optional = true } uuid = { version = "1", features = ["v4", "v7"] } chrono = { version = "0.4", default-features = false, features = ["serde", "clock"] } thiserror = "2" diff --git a/crates/uteke-core/src/edges.rs b/crates/uteke-core/src/edges.rs index ad708627..814c0bb4 100644 --- a/crates/uteke-core/src/edges.rs +++ b/crates/uteke-core/src/edges.rs @@ -1995,9 +1995,10 @@ impl crate::Uteke { .ok_or_else(|| Error::validation(format!("New memory not found: {new_id}")))?; let now = chrono::Utc::now().to_rfc3339(); - let reason_text = reason - .map(str::to_string) - .unwrap_or_else(|| format!("superseded by {new_id}")); + let reason_text = match reason { + Some(r) => format!("superseded by {new_id}: {r}"), + None => format!("superseded by {new_id}"), + }; let conn = self.graph_store(); // Edge pair AND the soft-deprecation in ONE transaction — a failure // in any of the three writes rolls the whole supersession back @@ -2035,10 +2036,53 @@ impl crate::Uteke { params![now, reason_text, old.id], ) .map_err(|e| Error::db("deprecate superseded memory", e))?; + // Re-supersession of an already-deprecated memory: refresh the + // reason/timestamp so the resolution ledger reflects the CURRENT + // winner (stale "superseded by " text would make the + // ledger contradict the edge pair — cora finding, #1172 F2). + tx.execute( + "UPDATE memories SET deprecate_reason = ?1, updated_at = ?2, deprecated_at = ?2 WHERE id = ?3 AND deprecated = 1", + params![reason_text, now, old.id], + ) + .map_err(|e| Error::db("refresh superseded reason", e))?; tx.commit() .map_err(|e| Error::db("commit supersede tx", e))?; } + // Provenance chain (#1172 Fase 2): record the resolution as + // provenance-bearing events on BOTH sides — the retired memory gets + // `superseded` (why it lost), the winner gets `updated` pointing + // back (what it replaced). Best-effort, like all timeline writes. + self.store + .add_timeline_event_with_provenance( + &old.id, + crate::timeline::TimelineEventType::Superseded, + Some(&serde_json::json!({ + "superseded_by": new.id, + "reason": reason_text, + })), + Some("system:supersede"), + Some(&serde_json::json!([ + {"memory": new.id, "relation": "supersedes_this"}, + {"memory": old.id, "relation": "subject"}, + ])), + ) + .unwrap_or_else(|e| tracing::warn!("superseded event failed for {}: {e}", old.id)); + self.store + .add_timeline_event_with_provenance( + &new.id, + crate::timeline::TimelineEventType::Updated, + Some(&serde_json::json!({ + "superseded": old.id, + "reason": reason_text, + })), + Some("system:supersede"), + Some(&serde_json::json!([ + {"memory": old.id, "relation": "superseded_by_this"}, + ])), + ) + .unwrap_or_else(|e| tracing::warn!("supersedes event failed for {}: {e}", new.id)); + // Same hygiene soft_forget() applies to deprecated rows: remove from // the vector index (deprecated memories must not surface in // semantic recall — code-scanning #685) and invalidate the recall @@ -2083,6 +2127,169 @@ impl crate::Uteke { })?; Ok(target) } + + /// Undo a supersession (#1172 Fase 2): restore the deprecated memory to + /// active, remove the supersession edge pair, and record + /// `supersession_undone` provenance events on both memories. + /// + /// Returns `Ok(None)` when the memory has no supersession to undo; + /// `Ok(Some(undone_id))` on success. The previously-winning memory stays + /// untouched and active — both now live side by side, with the full + /// history auditable via provenance. + pub fn undo_supersession(&self, memory_id: &str) -> Result, Error> { + let superseded_by = match self.supersession_of(memory_id)? { + Some(new_id) => new_id, + None => return Ok(None), + }; + + let now = chrono::Utc::now().to_rfc3339(); + let conn = self.graph_store(); + // Restore AND edge-pair removal in ONE transaction — a failure in any + // write rolls the whole undo back. Committing the restore first would + // leave a live superseded_by pair on an ACTIVE memory if the edge + // deletes failed (the exact partial state the supersede tx comment + // warns about) — cora finding on the #1172 F2 branch. + { + let tx = conn + .unchecked_transaction() + .map_err(|e| Error::db("begin undo_supersession tx", e))?; + tx.execute( + "UPDATE memories SET deprecated = 0, valid_until = NULL, deprecate_reason = NULL, deprecated_at = NULL, updated_at = ?1 WHERE id = ?2 AND deprecated = 1", + params![now, memory_id], + ) + .map_err(|e| Error::db("restore superseded memory", e))?; + tx.execute( + "DELETE FROM memory_edges WHERE source_id = ?1 AND edge_type = ?2 AND target_id = ?3", + params![memory_id, EDGE_SUPERSEDED_BY, superseded_by], + ) + .map_err(|e| Error::db("delete superseded_by edge", e))?; + tx.execute( + "DELETE FROM memory_edges WHERE source_id = ?1 AND edge_type = ?2 AND target_id = ?3", + params![superseded_by, EDGE_SUPERSEDES, memory_id], + ) + .map_err(|e| Error::db("delete supersedes edge", e))?; + tx.commit() + .map_err(|e| Error::db("commit undo_supersession tx", e))?; + } + + // Post-commit side effects (same contract as promote()): re-add to + // the vector index and invalidate the recall cache. + if let Some(memory) = self.store.get_by_id(memory_id).ok().flatten() { + if !memory.embedding.is_empty() { + let mut index = self + .index + .write() + .map_err(|_| Error::lock("index write lock during undo_supersession"))?; + if let Err(e) = index.insert(&memory.id, &memory.embedding) { + tracing::warn!( + "Failed to re-insert memory id={} into vector index during undo_supersession: {e}", + memory.id + ); + } + let _ = index.save(); + } + self.recall_cache.invalidate_namespace(&memory.namespace); + } + + // Provenance events on both sides — the undo itself is auditable. + self.store + .add_timeline_event_with_provenance( + memory_id, + crate::timeline::TimelineEventType::SupersessionUndone, + Some(&serde_json::json!({ + "was_superseded_by": superseded_by, + })), + Some("system:undo_supersession"), + Some(&serde_json::json!([ + {"memory": superseded_by, "relation": "was_superseded_by"}, + ])), + ) + .unwrap_or_else(|e| { + tracing::warn!("supersession_undone event failed for {memory_id}: {e}") + }); + self.store + .add_timeline_event_with_provenance( + &superseded_by, + crate::timeline::TimelineEventType::Updated, + Some(&serde_json::json!({ + "supersession_of": memory_id, + "undone": true, + })), + Some("system:undo_supersession"), + Some(&serde_json::json!([ + {"memory": memory_id, "relation": "supersession_undone"}, + ])), + ) + .unwrap_or_else(|e| tracing::warn!("undo event failed for {superseded_by}: {e}")); + + Ok(Some(superseded_by.to_string())) + } + + /// List memories in a namespace that were superseded but not restored + /// (#1172 Fase 2) — the auditable resolution ledger: each entry carries + /// the retired memory, its winner (via provenance), and the reason. + /// Membership is EDGE-driven: a row is listed iff it is deprecated AND + /// carries a `superseded_by` edge — the exact predicate + /// `undo_supersession` resolves against. Free-text reason matching would + /// diverge (legacy custom reasons invisible; forget-reasons that mention + /// "superseded" falsely listed) — cora finding on the #1172 F2 branch. + pub fn contradiction_resolutions( + &self, + namespace: Option<&str>, + limit: usize, + ) -> Result, Error> { + let sql = if namespace.is_some() { + "SELECT m.id, m.content, m.memory_type, m.namespace, m.tags, m.importance, m.deprecated_at, m.deprecate_reason \ + FROM memories m \ + JOIN memory_edges e ON e.source_id = m.id AND e.edge_type = ?2 \ + WHERE m.deprecated = 1 AND m.namespace = ?1 \ + ORDER BY m.deprecated_at DESC LIMIT ?3" + } else { + "SELECT m.id, m.content, m.memory_type, m.namespace, m.tags, m.importance, m.deprecated_at, m.deprecate_reason \ + FROM memories m \ + JOIN memory_edges e ON e.source_id = m.id AND e.edge_type = ?1 \ + WHERE m.deprecated = 1 \ + ORDER BY m.deprecated_at DESC LIMIT ?2" + }; + let mut stmt = self + .graph_store() + .prepare(sql) + .map_err(|e| Error::db("prepare contradiction_resolutions", e))?; + let map_row = |r: &rusqlite::Row| -> rusqlite::Result { + let tags_str: String = r.get(4)?; + let tags = serde_json::from_str::>(&tags_str).unwrap_or_default(); + Ok(crate::DeprecatedMemoryInfo { + id: r.get(0)?, + content: r.get(1)?, + memory_type: r.get(2)?, + namespace: r.get(3)?, + tags, + importance: r.get(5).unwrap_or(0.5), + deprecated_at: r + .get::<_, Option>(6)? + .and_then(|s| chrono::DateTime::parse_from_rfc3339(&s).ok()) + .map(|dt| dt.with_timezone(&chrono::Utc)), + deprecate_reason: r.get(7)?, + }) + }; + let rows = match namespace { + Some(ns) => stmt + .query_map(params![ns, EDGE_SUPERSEDED_BY, limit as i64], map_row) + .map_err(|e| Error::db("query contradiction_resolutions", e))?, + // Both variants bind every placeholder positionally (?1..?n). + // (The original no-ns variant declared LIMIT ?2 while binding a + // single param → runtime bind error, caught by the #1172 F2 MCP + // roundtrip test.) + None => stmt + .query_map(params![EDGE_SUPERSEDED_BY, limit as i64], map_row) + .map_err(|e| Error::db("query contradiction_resolutions", e))?, + }; + let mut out = Vec::new(); + for row in rows { + out.push(row.map_err(|e| Error::db("contradiction_resolutions row", e))?); + } + Ok(out) + } } #[cfg(test)] @@ -2204,4 +2411,180 @@ mod supersession_tests { drop(uteke); std::fs::remove_dir_all(&dir).ok(); } + + /// #1172 Fase 2: supersede writes provenance-bearing events on both + /// sides; contradiction_resolutions lists the retired memory; undo + /// restores it, removes the pair, and records supersession_undone. + #[test] + fn supersession_evidence_chain_and_undo() { + let dir = std::env::temp_dir().join(format!("chain-{}", std::process::id())); + std::fs::create_dir_all(&dir).unwrap(); + let uteke = Uteke::open(dir.join("t.db").to_str().unwrap()).unwrap(); + let embedding = vec![0.6_f32; 768]; + + let old = uteke + .remember_precomputed( + "old claim: deploy at 8am", + &[], + None, + Some("chain-ns"), + "fact", + "text", + &embedding, + ) + .unwrap(); + let new = uteke + .remember_precomputed( + "new claim: deploy at 9am", + &[], + None, + Some("chain-ns"), + "fact", + "text", + &embedding, + ) + .unwrap(); + + uteke + .supersede(&old, &new, Some("corrected deploy time")) + .unwrap(); + + // The retired memory's provenance carries the superseded event with + // actor + evidence pointing at the winner. + let report = uteke.provenance(&old).unwrap().expect("old exists"); + let superseded = report + .events + .iter() + .find(|e| e.event_type == "superseded") + .expect("superseded event recorded"); + assert_eq!(superseded.actor.as_deref(), Some("system:supersede")); + let evidence = superseded.evidence.as_ref().expect("evidence"); + assert_eq!(evidence[0]["memory"], serde_json::json!(new)); + // Supersede always sets a deprecation reason — verified via the + // superseded event's reason payload. + assert!( + superseded.event_data.is_some(), + "superseded event must carry the reason payload" + ); + + // The winner's chain carries a mirrored updated event. + let new_report = uteke.provenance(&new).unwrap().expect("new exists"); + assert!( + new_report.events.iter().any( + |e| e.event_type == "updated" && e.actor.as_deref() == Some("system:supersede") + ) + ); + + // The resolution ledger lists the retired memory. + let ledger = uteke + .contradiction_resolutions(Some("chain-ns"), 50) + .unwrap(); + assert!( + ledger.iter().any(|d| d.id == old), + "retired memory in ledger" + ); + + // Undo: restores the old memory, removes the pair, records events. + let undone = uteke.undo_supersession(&old).unwrap(); + assert_eq!(undone.as_deref(), Some(new.as_str())); + assert!( + uteke.undo_supersession(&old).unwrap().is_none(), + "second undo is a no-op" + ); + + let restored = uteke.provenance(&old).unwrap().expect("old exists"); + assert!(!restored.deprecated, "undo must restore the memory"); + assert!( + restored + .events + .iter() + .any(|e| e.event_type == "supersession_undone"), + "supersession_undone event recorded" + ); + assert_eq!( + uteke.supersession_of(&old).unwrap(), + None, + "supersession pair removed" + ); + + // Ledger no longer contains the restored memory. + let ledger_after = uteke + .contradiction_resolutions(Some("chain-ns"), 50) + .unwrap(); + assert!(!ledger_after.iter().any(|d| d.id == old)); + + drop(uteke); + std::fs::remove_dir_all(&dir).ok(); + } +} + +#[cfg(test)] +mod resupersession_ledger_tests { + use crate::Uteke; + + /// Regression (cora finding, #1172 F2): re-superseding an already- + /// deprecated memory must refresh deprecate_reason/deprecated_at so the + /// resolution ledger names the CURRENT winner, not the first one. + #[test] + fn resupersession_refreshes_ledger_reason() { + let dir = std::env::temp_dir().join(format!( + "rsled-{}-{}", + std::process::id(), + std::time::SystemTime::now() + .duration_since(std::time::UNIX_EPOCH) + .map(|d| d.as_nanos()) + .unwrap_or(0) + )); + std::fs::create_dir_all(&dir).unwrap(); + let uteke = Uteke::open(dir.join("t.db").to_str().unwrap()).unwrap(); + + let embedding = vec![0.5_f32; 768]; + let mk = |content: &str| -> String { + uteke + .remember_precomputed( + content, + &[], + None, + Some("rs-ns"), + "decision", + "text", + &embedding, + ) + .unwrap() + }; + let old = mk("old decision"); + let mid = mk("mid decision"); + let latest = mk("latest decision"); + + uteke.supersede(&old, &mid, Some("first pick")).unwrap(); + // Re-supersession: old is already deprecated; winner changes. + uteke + .supersede(&old, &latest, Some("second pivot")) + .unwrap(); + + // Edge pointer is the current winner. + assert_eq!( + uteke.supersession_of(&old).unwrap().as_deref(), + Some(latest.as_str()) + ); + + // Ledger must reflect the CURRENT winner and reason. + let ledger = uteke.contradiction_resolutions(Some("rs-ns"), 50).unwrap(); + let entry = ledger + .iter() + .find(|d| d.id == old) + .expect("old memory in ledger"); + let reason = entry.deprecate_reason.as_deref().unwrap_or(""); + assert!( + reason.contains(latest.as_str()) && reason.contains("second pivot"), + "ledger reason must name the current winner: {reason}" + ); + assert!( + !reason.contains(mid.as_str()), + "stale first-winner reason must be gone: {reason}" + ); + + drop(uteke); + std::fs::remove_dir_all(&dir).ok(); + } } diff --git a/crates/uteke-core/src/graph.rs b/crates/uteke-core/src/graph.rs index ed649a4f..52eaf5cc 100644 --- a/crates/uteke-core/src/graph.rs +++ b/crates/uteke-core/src/graph.rs @@ -297,6 +297,104 @@ impl<'a> GraphStore<'a> { Ok(id) } + /// Find the graph node representing a memory, if any (read-only). + /// + /// Resolution order (#1180): + /// 1. Node linked via `graph_nodes.memory_id` (authoritative). + /// 2. Node whose label equals the memory ID — legacy rows created by + /// older servers that inserted the raw memory ID as `source`/`target` + /// before the FK mismatch was fixed. + pub fn node_id_for_memory(&self, memory_id: &str) -> Result, Error> { + let linked: Option = self + .conn + .query_row( + "SELECT id FROM graph_nodes WHERE memory_id = ?1 LIMIT 1", + params![memory_id], + |row| row.get(0), + ) + .optional() + .map_err(|e| Error::db("graph node lookup by memory", e))?; + if linked.is_some() { + return Ok(linked); + } + self.find_node(memory_id)? + .map(|node| Ok(node.id)) + .transpose() + } + + /// Find or create the graph node representing a memory. Returns node ID. + /// + /// Resolves via [`Self::node_id_for_memory`]; creates a node with + /// `label = memory_id`, `entity_type = "memory"`, and the `memory_id` + /// link set when none exists. This is what makes `POST /graph/edge` + /// work with plain memory IDs — `graph_edges.source_id/target_id` have + /// FKs to `graph_nodes(id)`, so memory IDs must be mapped to nodes + /// before insertion (#1180). + /// + /// #1187: newly created memory nodes carry a readable content preview + /// (first 60 chars) as their label instead of the raw memory UUID. + /// Legacy rows whose label is still the raw memory ID are upgraded + /// in place here. + pub fn ensure_node_for_memory(&self, memory_id: &str) -> Result { + let node_id = match self.node_id_for_memory(memory_id)? { + Some(id) => id, + None => { + let label = self.memory_node_label(memory_id)?; + return self.upsert_node(&label, Some("memory"), Some(memory_id)); + } + }; + + // #1187 legacy upgrade: raw-UUID labels → readable preview labels. + // Entity nodes and non-UUID labels are untouched. + let label_is_memory_id: bool = self + .conn + .query_row( + "SELECT COUNT(*) FROM graph_nodes WHERE id = ?1 AND label = ?2 AND memory_id = ?2 AND entity_type = 'memory'", + params![node_id, memory_id], + |row| row.get::<_, i64>(0), + ) + .map(|c| c > 0) + .map_err(|e| Error::db("graph node label probe", e))?; + if label_is_memory_id { + let label = self.memory_node_label(memory_id)?; + self.conn + .execute( + "UPDATE graph_nodes SET label = ?1 WHERE id = ?2", + params![label, node_id], + ) + .map_err(|e| Error::db("graph node label upgrade", e))?; + } + Ok(node_id) + } + + /// Readable node label for a memory (#1187): `preview — <8-char id>` + /// so two memories with identical 60-char prefixes still get distinct + /// labels (upsert is label-keyed; a bare preview could collide or + /// hijack an existing entity node). + fn memory_node_label(&self, memory_id: &str) -> Result { + let preview = self.memory_label_preview(memory_id)?; + let short_id: String = memory_id.chars().take(8).collect(); + Ok(match preview { + Some(p) => format!("{p} — {short_id}"), + None => memory_id.to_string(), + }) + } + + /// Content preview for a memory's node label (#1187): first 60 chars of + /// the memory content. `None` when the memory does not exist. + fn memory_label_preview(&self, memory_id: &str) -> Result, Error> { + let row: Option = self + .conn + .query_row( + "SELECT substr(content, 1, 60) FROM memories WHERE id = ?1", + params![memory_id], + |row| row.get(0), + ) + .optional() + .map_err(|e| Error::db("memory label preview", e))?; + Ok(row.filter(|s| !s.trim().is_empty())) + } + /// Create an edge between two nodes. Ignores if edge already exists (INSERT OR IGNORE). pub fn add_edge( &self, @@ -653,9 +751,14 @@ mod tests { fn setup() -> Connection { let conn = Connection::open_in_memory().unwrap(); // Don't enable foreign_keys in test — graph_nodes references memories(id) - // which doesn't exist in this minimal test setup. + // which doesn't exist in this minimal test setup. A minimal memories + // table IS present so #1187 label previews can resolve. conn.execute_batch( r#" + CREATE TABLE memories ( + id TEXT PRIMARY KEY, + content TEXT NOT NULL DEFAULT '' + ); CREATE TABLE graph_nodes ( id TEXT PRIMARY KEY, label TEXT NOT NULL COLLATE NOCASE, @@ -708,6 +811,76 @@ mod tests { assert_eq!(id1, id2); } + // ── Memory→node resolution (#1180) ───────────────────────────────── + + #[test] + fn test_node_id_for_memory_prefers_linked_node() { + let conn = setup(); + let g = GraphStore::new(&conn); + let mid = "01890a5d-ac96-774b-bcce-b30220900000"; + let linked = g.upsert_node("Alice", Some("memory"), Some(mid)).unwrap(); + assert_eq!(g.node_id_for_memory(mid).unwrap(), Some(linked)); + } + + #[test] + fn test_node_id_for_memory_falls_back_to_legacy_label() { + let conn = setup(); + let g = GraphStore::new(&conn); + // Legacy rows from pre-#1180 servers carried the raw memory ID as + // source/target, so a node with label == memory ID may already exist. + let mid = "legacy-memory-id"; + let legacy = g.upsert_node(mid, None, None).unwrap(); + assert_eq!(g.node_id_for_memory(mid).unwrap(), Some(legacy)); + } + + #[test] + fn test_node_id_for_memory_none_when_absent() { + let conn = setup(); + let g = GraphStore::new(&conn); + assert_eq!(g.node_id_for_memory("ghost-id").unwrap(), None); + } + + #[test] + fn test_ensure_node_for_memory_creates_linked_node_idempotently() { + let conn = setup(); + conn.execute( + "INSERT INTO memories (id, content) VALUES ('01890a5d-ac96-774b-bcce-b30220900001', 'Deploy uses Bun, never npm')", + [], + ) + .unwrap(); + let g = GraphStore::new(&conn); + let mid = "01890a5d-ac96-774b-bcce-b30220900001"; + let first = g.ensure_node_for_memory(mid).unwrap(); + let second = g.ensure_node_for_memory(mid).unwrap(); + assert_eq!(first, second, "ensure must be idempotent"); + let node = g.get_node(&first).unwrap().expect("node must exist"); + // #1187: label is a readable content preview + short id, not the UUID. + assert_eq!(node.label, "Deploy uses Bun, never npm — 01890a5d"); + assert_eq!(node.entity_type.as_deref(), Some("memory")); + assert_eq!(node.memory_id.as_deref(), Some(mid)); + } + + #[test] + fn test_ensure_node_for_memory_reuses_existing_linked_node() { + let conn = setup(); + let g = GraphStore::new(&conn); + let mid = "mem-1"; + let existing = g.upsert_node("Alice", Some("person"), Some(mid)).unwrap(); + assert_eq!(g.ensure_node_for_memory(mid).unwrap(), existing); + } + + #[test] + fn test_ensure_node_for_memory_edge_roundtrip() { + let conn = setup(); + let g = GraphStore::new(&conn); + let a = g.ensure_node_for_memory("mem-a").unwrap(); + let b = g.ensure_node_for_memory("mem-b").unwrap(); + g.add_edge(&a, &b, "related", 1.0).unwrap(); + let edges = g.neighbors(&a, 1).unwrap(); + assert_eq!(edges.len(), 1); + assert_eq!(edges[0].target_id, b); + } + #[test] fn test_add_edge() { let conn = setup(); @@ -894,4 +1067,152 @@ mod tests { assert_eq!(nodes.len(), 3); assert_eq!(nodes[0].label, "Alice"); // sorted } + + // ── #1187 legacy label upgrade + #1189 graph liveness ────────────── + + #[test] + fn test_ensure_node_for_memory_upgrades_legacy_uuid_label() { + let conn = setup(); + conn.execute( + "INSERT INTO memories (id, content) VALUES ('legacy-mem-1', 'Vector index uses usearch by default')", + [], + ) + .unwrap(); + let g = GraphStore::new(&conn); + // Legacy row: raw memory UUID as the label (pre-#1187 servers). + let legacy_id = g + .upsert_node("legacy-mem-1", Some("memory"), Some("legacy-mem-1")) + .unwrap(); + + let node = g.get_node(&legacy_id).unwrap().expect("node exists"); + assert_eq!( + node.label, "legacy-mem-1", + "precondition: legacy UUID label" + ); + + // First ensure() must upgrade the label in place, keeping the ID. + let id = g.ensure_node_for_memory("legacy-mem-1").unwrap(); + assert_eq!(id, legacy_id, "node id must be preserved"); + let node = g.get_node(&id).unwrap().expect("node exists"); + assert_eq!( + node.label, "Vector index uses usearch by default — legacy-m", + "label upgraded to readable preview" + ); + } + + #[test] + fn test_graph_data_filters_stale_memory_nodes() { + let uteke = crate::Uteke::open(":memory:").unwrap(); + // Embedder-free seeding (CI has no ONNX runtime) — the graph layer + // under test is storage-only, vectors are opaque bytes here. + let embedding = vec![0.1_f32; 768]; + let keep = uteke + .remember_precomputed( + "active memory about caching", + &[], + None, + Some("gns"), + "fact", + "text", + &embedding, + ) + .unwrap(); + let gone = uteke + .remember_precomputed( + "memory that will be forgotten", + &[], + None, + Some("gns"), + "fact", + "text", + &embedding, + ) + .unwrap(); + let dep = uteke + .remember_precomputed( + "memory that will be deprecated", + &[], + None, + Some("gns"), + "fact", + "text", + &embedding, + ) + .unwrap(); + uteke.forget(&gone).unwrap(); + uteke.supersede(&dep, &keep, Some("pivot")).unwrap(); + + let gs = crate::GraphStore::new(uteke.graph_store()); + // One node per live memory + the stale ones (the bug setup). + let n_keep = gs.ensure_node_for_memory(&keep).unwrap(); + let n_gone = gs.ensure_node_for_memory(&gone).unwrap(); + let n_dep = gs.ensure_node_for_memory(&dep).unwrap(); + gs.add_edge(&n_keep, &n_dep, "related", 1.0).unwrap(); + gs.add_edge(&n_dep, &n_gone, "related", 1.0).unwrap(); + + // No namespace: stale nodes (forgotten + deprecated) must vanish. + let data = uteke.graph_data(None).unwrap(); + let ids: Vec<&str> = data.nodes.iter().map(|n| n.id.as_str()).collect(); + assert!( + !ids.contains(&n_gone.as_str()), + "forgotten memory's node must be filtered (#1189)" + ); + assert!( + !ids.contains(&n_dep.as_str()), + "deprecated memory's node must be filtered (#1189)" + ); + assert!( + ids.contains(&n_keep.as_str()), + "live memory's node must stay" + ); + // Edges touching removed nodes are gone too. + assert_eq!( + data.edges.len(), + 0, + "all edges touched stale nodes: {:?}", + data.edges + ); + // Stats reflect the filtered graph. + assert_eq!(data.stats.node_count, ids.len()); + } + + #[test] + fn test_graph_data_namespace_filter_still_excludes_other_ns() { + let uteke = crate::Uteke::open(":memory:").unwrap(); + let embedding = vec![0.1_f32; 768]; + let in_ns = uteke + .remember_precomputed( + "in target namespace", + &[], + None, + Some("ns-a"), + "fact", + "text", + &embedding, + ) + .unwrap(); + let other = uteke + .remember_precomputed( + "in another namespace", + &[], + None, + Some("ns-b"), + "fact", + "text", + &embedding, + ) + .unwrap(); + let gs = crate::GraphStore::new(uteke.graph_store()); + let _ = gs.ensure_node_for_memory(&in_ns).unwrap(); + let _ = gs.ensure_node_for_memory(&other).unwrap(); + + let data = uteke.graph_data(Some("ns-a")).unwrap(); + let mids: Vec<&str> = data + .nodes + .iter() + .filter_map(|n| n.memory_id.as_deref()) + .collect(); + assert!(mids.contains(&in_ns.as_str())); + assert!(!mids.contains(&other.as_str()), "other ns must be excluded"); + } } diff --git a/crates/uteke-core/src/lib.rs b/crates/uteke-core/src/lib.rs index c3d9509c..7c3535b0 100644 --- a/crates/uteke-core/src/lib.rs +++ b/crates/uteke-core/src/lib.rs @@ -31,6 +31,7 @@ mod operations; mod orphans; pub mod provenance; mod recall_cache; +pub mod recall_explain; mod rooms; pub mod rooms_segments; pub mod salience_recency; @@ -54,9 +55,9 @@ pub use graph_rerank::{GraphRerankConfig, GraphSignals, compute_graph_signals, r pub use memory::aging::DeprecatedMemoryInfo; pub use memory::types::{ AgingStatus, BulkDeleteResult, CleanupResult, ConsolidationResult, ContradictionResult, - DEFAULT_NAMESPACE, ExportEntry, ImportResult, Memory, MemoryTier, MemoryType, PruneResult, - RecallStrategy, SearchResult, SearchResultType, SearchType, SimilarPair, StoreStats, TagInfo, - UnifiedSearchResult, + DEFAULT_NAMESPACE, ExportEntry, ImportResult, Memory, MemoryTier, MemoryType, + NamespaceDeleteResult, NamespaceRenameResult, PruneResult, RecallStrategy, SearchResult, + SearchResultType, SearchType, SimilarPair, StoreStats, TagInfo, UnifiedSearchResult, }; pub use memory::{ DocumentEntry, DocumentSection, Room, RoomDocument, RoomMemory, RoomStats, RoomSummary, @@ -64,6 +65,7 @@ pub use memory::{ documents::{Document, DocumentChunk, DocumentSearchResult, DocumentSummary}, }; pub use orphans::{DEFAULT_ORPHAN_THRESHOLD, OrphanMemory, compute_orphan_score}; +pub use provenance::{ProvenanceReport, TrustTier}; pub use salience_recency::{ SalienceRecencyConfig, apply_boosts, recency_score, salience_score, type_half_life_days, }; @@ -583,6 +585,23 @@ pub struct Uteke { embed_cache_path: Option, } +/// Canonical uteke embedding width (EmbeddingGemma ONNX, 768). +/// +/// Last-resort fallback when opening a store with no embedder, no +/// UTEKE_EMBEDDING_DIMS, and no persisted embeddings (#1166). +const DEFAULT_EMBEDDING_DIMS: usize = 768; + +/// Default embedder backend for `Uteke::open` (#1166). +/// +/// Feature-aware: a build compiled without the `onnx` feature can never +/// initialize the onnx backend, so the default falls back to `""` (no +/// embedder configured; lazy init is skipped until an embedding arrives +/// via the injected-embedding path). With `onnx` compiled in, the default +/// stays `"onnx"` as before. +fn default_embedder_backend() -> &'static str { + if cfg!(feature = "onnx") { "onnx" } else { "" } +} + impl Uteke { /// Borrow the underlying store (for CLI/advanced use). pub fn store(&self) -> &Store { @@ -648,7 +667,60 @@ impl Uteke { Self::finish_open( store, None, - "onnx".to_string(), + default_embedder_backend().to_string(), + TierConfig::default(), + RecallConfig::default(), + EmbeddingSettings::default(), + ) + } + + /// Resolve the runtime vector-engine preference (#1168): + /// `UTEKE_VECTOR_BACKEND` env → caller-supplied (uteke.toml) → default. + /// Returns None when unset/empty/unknown (logged) or not compiled in. + fn resolve_vector_backend_pref( + config_pref: Option<&str>, + ) -> Option { + use memory::vector::VectorBackend; + + let raw = std::env::var("UTEKE_VECTOR_BACKEND") + .ok() + .filter(|v| !v.is_empty()) + .or_else(|| config_pref.filter(|v| !v.is_empty()).map(|v| v.to_string())); + + let raw = raw?; + match VectorBackend::parse(&raw) { + Some(b) if b.is_compiled_in() => Some(b), + Some(b) => { + tracing::warn!( + "UTEKE_VECTOR_BACKEND='{}' requested engine '{b:?}' is not compiled in; falling back to the default engine", + raw + ); + None + } + None => { + tracing::warn!( + "Invalid vector backend '{raw}' (expected 'usearch' or 'vecq'); using the default engine" + ); + None + } + } + } + + /// Open with an explicit embedder backend choice (#1166). + /// + /// `Some("onnx" | "openai" | "ollama")` selects the backend lazily, exactly + /// like `open_with_embedding_and_graph`. `None` opens the store **without + /// an embedder**: FTS5 keyword recall works, and embeddings can be + /// supplied per-write by hosts that embed externally (mobile profiles, + /// injected-embedding pipelines). Useful for builds compiled without the + /// `onnx` feature, where the old `open()` default requested a backend + /// that could never initialize. + pub fn open_with_backend(path: impl AsRef, backend: Option<&str>) -> Result { + let (_db_str, store) = Self::open_store(path)?; + Self::finish_open( + store, + None, + backend.unwrap_or("").to_string(), TierConfig::default(), RecallConfig::default(), EmbeddingSettings::default(), @@ -665,6 +737,7 @@ impl Uteke { tier_config: TierConfig, recall_config: RecallConfig, graph_rerank_config: graph_rerank::GraphRerankConfig, + vector_backend: Option<&str>, ) -> Result { let (_db_str, store) = Self::open_store(path)?; Self::finish_open_full( @@ -675,6 +748,7 @@ impl Uteke { recall_config, settings, graph_rerank_config, + vector_backend, ) } @@ -701,9 +775,11 @@ impl Uteke { recall_config, embedding_settings, graph_rerank::GraphRerankConfig::default(), + None, ) } + #[allow(clippy::too_many_arguments)] fn finish_open_full( store: Store, embedder: Option>, @@ -712,7 +788,11 @@ impl Uteke { recall_config: RecallConfig, embedding_settings: EmbeddingSettings, graph_rerank_config: graph_rerank::GraphRerankConfig, + vector_backend_pref: Option<&str>, ) -> Result { + // Runtime engine selection (#1168): ENV > config > compiled-in default. + let vector_backend = Self::resolve_vector_backend_pref(vector_backend_pref) + .unwrap_or_else(memory::vector::VectorBackend::default_backend); // Determine index path: same directory as SQLite DB. // `:memory:` databases get an EPHEMERAL in-memory index instead: // persisting to `./uteke_index.usearch` in the CWD would make every @@ -725,11 +805,15 @@ impl Uteke { None } else { let dir = p.parent().unwrap_or(Path::new(".")); - // Backend-specific extension (#1112): a vecq build reads/writes - // `uteke_index.vecq`, a usearch build `uteke_index.usearch`. - // Cross-backend opens no longer parse (and re-save over) the - // other format's file. - Some(dir.join(format!("uteke_index.{}", crate::memory::vector::INDEX_EXT))) + // Engine-specific extension (#1112, #1168): the selected + // engine reads/writes its own file and never parses (nor + // re-saves over) the other engine's file. Switching engines + // leaves the old file in place; the new engine starts empty + // and the rebuild-from-SQLite path below repopulates it. + Some(dir.join(format!( + "uteke_index.{}", + memory::vector::index_ext_for(vector_backend) + ))) } }); @@ -740,6 +824,12 @@ impl Uteke { None => match embedder_backend.as_str() { #[cfg(feature = "onnx")] "onnx" | "" | "custom" => crate::embed::OnnxEmbedder::dims(), + // No embedder configured (open_with_backend(.., None)) on a + // build without the onnx feature (#1166): dims stay unknown + // until the first externally-supplied embedding lands. The + // index starts empty and accepts the first injected dims. + #[cfg(not(feature = "onnx"))] + "" => 0, "openai" => { // User-configurable via uteke.toml or UTEKE_EMBEDDING_DIMS. // Default 1536 (text-embedding-3-small). @@ -766,8 +856,42 @@ impl Uteke { }, }; + // #1166: dims == 0 means "unknown yet" (open_with_backend(.., None) on + // a build without onnx). The vector index cannot be created at dims 0, + // so resolve from, in order: explicit UTEKE_EMBEDDING_DIMS, then + // embeddings already persisted in the store. Only if both are empty + // (fresh store, no embedder) fail with a clear error instead of a + // vecq/usearch panic on dims 0. + let dims = if dims == 0 { + let cfg = EmbeddingSettings::resolve_with_defaults(&embedding_settings); + if cfg.dims > 0 { + cfg.dims + } else if let Some(inferred) = store.infer_embedding_dims() { + tracing::debug!( + dims = inferred, + "Inferred embedding dims from existing store" + ); + inferred + } else { + // Nothing else is known: assume the canonical uteke dims + // (EmbeddingGemma ONNX, 768). This keeps plain open() working + // on builds compiled without the onnx feature (#1166) — the + // mobile/FFI profile embeds externally at 768 anyway. A host + // injecting embeddings of a different width gets a clear + // dimension-mismatch error at insert time and can rebuild the + // index via uteke repair. + tracing::warn!( + dims = DEFAULT_EMBEDDING_DIMS, + "No embedder, no UTEKE_EMBEDDING_DIMS, no existing embeddings; assuming canonical dims" + ); + DEFAULT_EMBEDDING_DIMS + } + } else { + dims + }; + let mut index = match &index_path { - Some(path) => match VectorIndex::load_or_create(path, dims) { + Some(path) => match VectorIndex::load_or_create_for(path, dims, vector_backend) { Ok(idx) => idx, Err(e) => { // Index file is corrupt (dim mismatch, truncated, etc). @@ -1207,6 +1331,15 @@ impl Uteke { self.store.set_source(id, source, source_type) } + /// Set the content hash recorded at write time (#1172 Fase 1). + /// + /// Normally set automatically by `remember*`/import paths; public so + /// repair/backfill tooling can populate legacy rows. Pass `None` to + /// clear. + pub fn set_source_hash(&self, id: &str, source_hash: Option<&str>) -> Result { + self.store.set_source_hash(id, source_hash) + } + /// Set author type on a memory (#1083): "human" | "agent". /// Invalid values return a Validation error. pub fn set_author_type(&self, id: &str, author_type: &str) -> Result { @@ -1231,18 +1364,34 @@ impl Uteke { /// Get graph nodes + edges for visualization (#408). /// /// Returns all nodes and edges in the knowledge graph, optionally - /// limited by namespace. + /// limited by namespace. Stale memory nodes are excluded in every + /// path (#1189): the namespace path matches `namespace AND + /// deprecated = 0`; the no-namespace path drops nodes whose parent + /// memory is missing (hard-deleted) or deprecated — so the graph + /// does not grow stale nodes on every forget/deprecate. pub fn graph_data(&self, namespace: Option<&str>) -> Result { let gs = GraphStore::new(&self.store.conn); let nodes = gs.all_nodes()?; let edges = gs.all_edges()?; let stats = gs.stats()?; + // Liveness set: memories that may appear in the graph — existing + // AND not deprecated (#1189). Legacy graph_rows whose memory was + // hard-deleted drop out here too (orphaned nodes). + let live_memory_ids: std::collections::HashSet = self + .store + .conn + .prepare("SELECT id FROM memories WHERE deprecated = 0") + .map_err(|e| Error::db("Failed to prepare liveness query", e))? + .query_map([], |row| row.get::<_, String>(0)) + .map_err(|e| Error::db("Failed to query live memories", e))? + .filter_map(|r| r.ok()) + .collect(); + // Filter by namespace if specified. // Memory-linked nodes are filtered by their parent memory's namespace. // Entity nodes (no memory_id) are always included (shared across namespaces). let (nodes, edges) = if let Some(ns) = namespace { - // Build a set of memory IDs that belong to this namespace. let ns_memory_ids: std::collections::HashSet = self .store .conn @@ -1278,7 +1427,33 @@ impl Uteke { (filtered_nodes, filtered_edges) } else { - (nodes, edges) + // #1189: no namespace → still drop nodes whose memory is gone + // or deprecated (the bug: this path returned all_nodes() raw). + let filtered_nodes: Vec = nodes + .into_iter() + .filter(|n| match &n.memory_id { + None => true, + Some(mid) => live_memory_ids.contains(mid), + }) + .collect(); + let node_ids: std::collections::HashSet<&str> = + filtered_nodes.iter().map(|n| n.id.as_str()).collect(); + let filtered_edges: Vec = edges + .into_iter() + .filter(|e| { + node_ids.contains(e.source_id.as_str()) + && node_ids.contains(e.target_id.as_str()) + }) + .collect(); + (filtered_nodes, filtered_edges) + }; + + // Stats count the FILTERED graph, not the raw tables (#1189) — + // otherwise GET /graph reported node counts that included stale rows. + let stats = crate::graph::GraphStats { + node_count: nodes.len(), + edge_count: edges.len(), + relation_types: stats.relation_types, }; Ok(GraphData { @@ -2408,6 +2583,202 @@ fn resolve_db_path(db_path: &Path) -> Result { #[cfg(test)] mod tests { + /// #1166: `open()` must not request a backend the build cannot + /// initialize. On a build without the `onnx` feature the default is + /// `""` (no embedder) — opening succeeds and FTS5 paths work without + /// ever touching an embedder. + #[test] + fn open_default_backend_is_feature_aware() { + // On onnx builds open() keeps working exactly as before. + if cfg!(feature = "onnx") { + let u = Uteke::open(":memory:").expect("open() must succeed with onnx feature"); + drop(u); + } else { + // Non-onnx build with an EMPTY store: no embedder, no persisted + // embeddings, so dims cannot be resolved. Must fail with a clear + // validation error (#1166) — never a vecq/usearch panic on 0 dims. + let _u = Uteke::open(":memory:") + .expect("open() must succeed on vecq-only builds via canonical-dims fallback"); + } + // The default must track the feature set in both directions. + assert_eq!( + default_embedder_backend(), + if cfg!(feature = "onnx") { "onnx" } else { "" } + ); + } + + /// #1166: explicit `None` backend opens without an embedder; a vecq-only + /// (no-onnx) build must open cleanly instead of failing on a hardcoded + /// "onnx" request. FTS5 write+search round-trip works without embedding. + #[test] + #[serial_test::serial] + fn open_with_backend_none_works_without_embedder() { + // With UTEKE_EMBEDDING_DIMS set, open succeeds on any build; writes + // store without embeddings and FTS search still finds them. + unsafe { std::env::set_var("UTEKE_EMBEDDING_DIMS", "768") }; + let u = Uteke::open_with_backend(":memory:", None) + .expect("open must succeed when dims are supplied via env"); + let id = u + .remember("sqlite fts fallback probe", &["probe"], None, None) + .expect("write without embedder must succeed"); + let _ = id; + let hits = u + .search("fts fallback", 5, None, None) + .expect("fts search must work without embedder"); + assert!( + !hits.is_empty(), + "FTS5 must find the probe without an embedder" + ); + unsafe { std::env::remove_var("UTEKE_EMBEDDING_DIMS") }; + } + + /// #1166: `open_with_backend(Some("openai"))` keeps the explicit-backend + /// contract (validation still happens at open, same as before). + #[test] + fn open_with_backend_invalid_name_rejected() { + let r = Uteke::open_with_backend(":memory:", Some("not-a-backend")); + let e = match r { + Err(e) => e.to_string(), + Ok(_) => panic!("invalid backend must fail at open()"), + }; + assert!( + e.contains("Unknown embedding backend"), + "expected unknown-backend error, got: {e}" + ); + } + + /// #1078 P0: ensure_embedder lazy-init branches — unknown backend + /// (rejected eagerly at open), custom backend without embedder + /// (immediate error), openai without API key (deterministic init error, + /// cached transient → identical message on 2nd call). + #[test] + fn ensure_embedder_unknown_backend_errors_not_cached() { + // Unknown backends are rejected eagerly at open() time (defensive: + // ensure_embedder re-checks, but open fails first with the same message). + let r = Uteke::open_with_embedding_and_graph( + ":memory:", + "not-a-backend", + EmbeddingSettings::default(), + TierConfig::default(), + RecallConfig::default(), + graph_rerank::GraphRerankConfig::default(), + None, + ); + let e = match r { + Err(e) => e.to_string(), + Ok(_) => panic!("unknown backend must fail at open()"), + }; + assert!( + e.contains("Unknown embedding backend: 'not-a-backend'"), + "got: {e}" + ); + } + + #[test] + fn ensure_embedder_custom_without_embedder_errors() { + // The "custom" backend dim-arm is gated on the onnx feature (it maps + // to the ONNX dims); on a vecq-only build open() rejects "custom" + // eagerly as unknown — both outcomes are correct, assert the one the + // current feature set produces (#1166). + if !cfg!(feature = "onnx") { + let r = Uteke::open_with_embedding_and_graph( + ":memory:", + "custom", + EmbeddingSettings::default(), + TierConfig::default(), + RecallConfig::default(), + graph_rerank::GraphRerankConfig::default(), + None, + ); + let e = match r { + Err(e) => e.to_string(), + Ok(_) => panic!("custom must be rejected at open() on non-onnx builds"), + }; + assert!( + e.contains("Unknown embedding backend: 'custom'"), + "got: {e}" + ); + return; + } + let u = Uteke::open_with_embedding_and_graph( + ":memory:", + "custom", + EmbeddingSettings::default(), + TierConfig::default(), + RecallConfig::default(), + graph_rerank::GraphRerankConfig::default(), + None, + ) + .unwrap(); + let e = u.embed_text("x").unwrap_err().to_string(); + assert!( + e.contains("Custom embedder backend set but no embedder was provided"), + "got: {e}" + ); + } + + #[test] + fn ensure_embedder_openai_missing_key_error_is_cached() { + // No UTEKE_EMBEDDING_API_KEY / OPENAI_API_KEY in env → deterministic + // init failure at OpenAiEmbedder::new, cached as transient (#822). + unsafe { + std::env::remove_var("UTEKE_EMBEDDING_API_KEY"); + std::env::remove_var("OPENAI_API_KEY"); + } + let u = Uteke::open_with_embedding_and_graph( + ":memory:", + "openai", + EmbeddingSettings::default(), + TierConfig::default(), + RecallConfig::default(), + graph_rerank::GraphRerankConfig::default(), + None, + ) + .unwrap(); + let e1 = u.embed_text("x").unwrap_err().to_string(); + assert!(e1.contains("requires an API key"), "got: {e1}"); + let e2 = u.embed_text("x").unwrap_err().to_string(); + assert!( + e2.contains("previously failed (cached)"), + "2nd call must hit the #822 cache: {e2}" + ); + } + + #[test] + fn validate_input_with_limits_matrix() { + use super::validate_input_with_limits; + + // 1) happy path + assert!(validate_input_with_limits("hello", &["a"], 100, 10, 20).is_ok()); + // 2) empty content (incl. whitespace-only) + assert!(validate_input_with_limits("", &["a"], 100, 10, 20).is_err()); + assert!(validate_input_with_limits(" \n\t", &["a"], 100, 10, 20).is_err()); + // 3) content exactly at limit → ok; over limit → err + let no_tags: &[&str] = &[]; + let at = "x".repeat(100); + assert!(validate_input_with_limits(&at, no_tags, 100, 10, 20).is_ok()); + let over = "x".repeat(101); + assert!(validate_input_with_limits(&over, no_tags, 100, 10, 20).is_err()); + // 4) content length check disabled when limit = 0 + let huge = "x".repeat(500); + assert!(validate_input_with_limits(&huge, no_tags, 0, 10, 20).is_ok()); + // 5) tags exactly at count limit → ok; over → err + let ten: Vec<&str> = vec!["t"; 10]; + assert!(validate_input_with_limits("c", &ten, 100, 10, 20).is_ok()); + let eleven: Vec<&str> = vec!["t"; 11]; + assert!(validate_input_with_limits("c", &eleven, 100, 10, 20).is_err()); + // 6) empty tag string rejected + assert!(validate_input_with_limits("c", &[""], 100, 10, 20).is_err()); + // 7) tag exactly at length limit → ok; over → err + let tag_at = "y".repeat(20); + assert!(validate_input_with_limits("c", &[&tag_at], 100, 10, 20).is_ok()); + let tag_over = "y".repeat(21); + assert!(validate_input_with_limits("c", &[&tag_over], 100, 10, 20).is_err()); + // 8) tag length check disabled when limit = 0 + let tag_huge = "y".repeat(200); + assert!(validate_input_with_limits("c", &[&tag_huge], 100, 10, 0).is_ok()); + } + #[ignore = "requires ONNX embedder — index file extension follows active backend"] #[test] fn index_file_uses_backend_extension() { @@ -2428,6 +2799,83 @@ mod tests { ); } + /// #1168 (dual-engine builds): switching the engine via preference + /// rebuilds from SQLite and both per-engine index files coexist. + /// Uses remember_precomputed-equivalent raw inserts so no embedder is + /// needed (works on any feature set with both engines compiled in). + #[cfg(all(feature = "usearch", feature = "vecq"))] + #[test] + #[serial_test::serial] + fn vector_engine_switch_rebuilds_from_sqlite() { + let dir = tempfile::tempdir().unwrap(); + let db = dir.path().join("uteke.db"); + + unsafe { std::env::set_var("UTEKE_VECTOR_BACKEND", "usearch") }; + let u = Uteke::open(&db).unwrap(); + u.remember_precomputed( + "engine switch probe", + &[], + None, + None, + "fact", + "text", + &vec![0.5f32; 768], + ) + .unwrap(); + u.shutdown().unwrap(); + drop(u); + assert!( + dir.path().join("uteke_index.usearch").exists(), + "usearch index file must exist" + ); + assert!( + !dir.path().join("uteke_index.vecq").exists(), + "vecq file must not exist yet" + ); + + // Switch to vecq: same store, engine preference flips. + unsafe { std::env::set_var("UTEKE_VECTOR_BACKEND", "vecq") }; + let u2 = Uteke::open(&db).unwrap(); + // finish_open_full auto-rebuilds the empty vecq index from SQLite. + let hits = u2 + .search("engine switch", 5, None, None) + .unwrap_or_default(); + assert!( + !hits.is_empty(), + "FTS5 must find the probe after engine switch" + ); + u2.shutdown().unwrap(); + drop(u2); + assert!( + dir.path().join("uteke_index.vecq").exists(), + "vecq index file must be written" + ); + // The old engine's file is left untouched (both coexist). + assert!( + dir.path().join("uteke_index.usearch").exists(), + "usearch file must survive the switch" + ); + + // And back to usearch — the old file is picked up, no rebuild needed. + unsafe { std::env::set_var("UTEKE_VECTOR_BACKEND", "usearch") }; + let u3 = Uteke::open(&db).unwrap(); + u3.shutdown().unwrap(); + drop(u3); + + unsafe { std::env::remove_var("UTEKE_VECTOR_BACKEND") }; + } + + /// #1168: invalid / not-compiled-in UTEKE_VECTOR_BACKEND falls back to the + /// default engine without failing the open. + #[test] + #[serial_test::serial] + fn vector_engine_env_invalid_falls_back() { + unsafe { std::env::set_var("UTEKE_VECTOR_BACKEND", "bogus-engine") }; + let u = Uteke::open(":memory:"); + assert!(u.is_ok(), "invalid engine name must fall back, not fail"); + unsafe { std::env::remove_var("UTEKE_VECTOR_BACKEND") }; + } + use super::*; use serial_test::serial; diff --git a/crates/uteke-core/src/maintenance.rs b/crates/uteke-core/src/maintenance.rs index ccd8b3da..20177ff9 100644 --- a/crates/uteke-core/src/maintenance.rs +++ b/crates/uteke-core/src/maintenance.rs @@ -182,17 +182,16 @@ impl crate::Uteke { /// Re-embed memories that have missing or empty embedding vectors. /// - /// Scans all non-deprecated memories, finds those with empty embeddings, - /// generates new embeddings, updates the database, and adds them to the index. + /// Scans active memories with NULL/empty embeddings via a dedicated SQL + /// query (`load_missing_embeddings`), generates new embeddings, updates + /// the database, and adds them to the index. pub fn reembed_missing(&self) -> Result { - let all_memories = self.store.load_all(None)?; - let total_scanned = all_memories.len(); - - // Filter to memories with empty embeddings, excluding deprecated. - let missing: Vec<&Memory> = all_memories - .iter() - .filter(|m| !m.deprecated && m.embedding.is_empty()) - .collect(); + // Scan directly for NULL/empty embeddings (#1146). This must NOT go + // through load_all(): its `embedding IS NOT NULL` guard (kept for + // index.build() safety, #992) filtered out exactly the rows this + // function exists to repair, making NULL rows permanently invisible. + let missing: Vec = self.store.load_missing_embeddings(None)?; + let total_scanned = self.store.load_all(None)?.len(); let missing_count = missing.len(); if missing_count == 0 { @@ -657,6 +656,91 @@ mod tests { assert_eq!(restored.deprecated, 3); } + #[ignore = "requires ONNX embedder — validates reembed repairs NULL-embedding rows (#1146)"] + #[test] + fn test_reembed_repairs_null_embedding_rows() { + // Regression test for #1146: `repair --reembed` used to scan through + // load_all(), whose `embedding IS NOT NULL` guard (#992) filtered out + // exactly the rows needing repair. NULL-embedding rows (write-path + // crash artifacts) were permanently invisible to reembed and doctor + // reported an unresolvable MISMATCH. + let dir = tempfile::tempdir().unwrap(); + let uteke = crate::Uteke::open(dir.path().join("uteke.db")).unwrap(); + + // Create one healthy memory (gets a real embedding). + let id = uteke + .remember("raft consensus requires a majority quorum", &[], None, None) + .unwrap(); + + // Simulate a write-path crash artifact: NULL embedding. + let id_null = uteke + .remember("vector quantization compresses embeddings", &[], None, None) + .unwrap(); + uteke + .graph_store() + .execute( + "UPDATE memories SET embedding = NULL WHERE id = ?1", + rusqlite::params![id_null], + ) + .unwrap(); + + // And the empty-blob variant (what the old code could only see). + let id_empty = uteke + .remember( + "hybrid search blends keyword and vector signals", + &[], + None, + None, + ) + .unwrap(); + uteke + .graph_store() + .execute( + "UPDATE memories SET embedding = X'' WHERE id = ?1", + rusqlite::params![id_empty], + ) + .unwrap(); + + // Scan finds both NULL and empty-blob rows. + let scanned = uteke.store.load_missing_embeddings(None).unwrap(); + assert_eq!( + scanned.len(), + 2, + "scan must see NULL-embedding rows, not just empty-blob ones" + ); + + // Reembed repairs both; the healthy row is untouched. + let report = uteke.reembed_missing().unwrap(); + assert_eq!(report.missing_count, 2); + assert_eq!(report.reembedded, 2, "both rows must be re-embedded"); + assert_eq!(report.failed, 0); + + // DB no longer has NULL/empty embeddings among active memories. + let remaining: i64 = uteke + .graph_store() + .query_row( + "SELECT COUNT(*) FROM memories WHERE (embedding IS NULL OR length(embedding) = 0) AND deprecated = 0", + [], + |r| r.get(0), + ) + .unwrap(); + assert_eq!(remaining, 0); + + // End-to-end: verify() reports a consistent store after repair. + let v = uteke.verify().unwrap(); + assert!( + v.consistent, + "store must be consistent after reembed, got db={} index={}", + v.db_count, v.index_count + ); + + // Reembed is now a no-op. + let again = uteke.reembed_missing().unwrap(); + assert_eq!(again.missing_count, 0); + assert_eq!(again.reembedded, 0); + let _ = (id, id_null, id_empty); + } + #[test] fn test_cleanup_result_serialization() { use crate::memory::types::CleanupResult; diff --git a/crates/uteke-core/src/memory/crud.rs b/crates/uteke-core/src/memory/crud.rs index 06a1d5e1..c2dae63c 100644 --- a/crates/uteke-core/src/memory/crud.rs +++ b/crates/uteke-core/src/memory/crud.rs @@ -634,6 +634,55 @@ impl super::Store { Ok(memories) } + /// Load active memories whose embedding is missing (SQL NULL or empty blob). + /// + /// Unlike [`load_all`], this deliberately includes NULL-embedding rows: it is + /// the scan source for `repair --reembed` (#1146). The NULL guard in + /// `load_all` exists to keep such rows out of `index.build()` (#992), but + /// reusing it as the reembed scan made NULL rows permanently invisible to + /// the one tool designed to fix them. + pub fn load_missing_embeddings(&self, namespace: Option<&str>) -> Result, Error> { + let sql = match namespace { + Some(_) => { + "SELECT id, content, embedding, tags, metadata, created_at, updated_at, namespace, access_count, last_accessed, deprecated, valid_from, valid_until, memory_type, importance, pinned, content_type, slug, source, source_type, author_type, deprecated_at FROM memories WHERE namespace = ?1 AND (embedding IS NULL OR length(embedding) = 0) AND deprecated = 0 ORDER BY created_at" + } + None => { + "SELECT id, content, embedding, tags, metadata, created_at, updated_at, namespace, access_count, last_accessed, deprecated, valid_from, valid_until, memory_type, importance, pinned, content_type, slug, source, source_type, author_type, deprecated_at FROM memories WHERE (embedding IS NULL OR length(embedding) = 0) AND deprecated = 0 ORDER BY created_at" + } + }; + + let mut memories = Vec::new(); + match namespace { + Some(ns) => { + let mut stmt = self + .conn + .prepare(sql) + .map_err(|e| Error::db("database operation", e))?; + let rows = stmt + .query_map(params![ns], row_to_memory) + .map_err(|e| Error::db("database store operation", e))?; + for row in rows { + let m = row.map_err(|e| Error::db("database operation", e))?; + memories.push(m); + } + } + None => { + let mut stmt = self + .conn + .prepare(sql) + .map_err(|e| Error::db("database operation", e))?; + let rows = stmt + .query_map([], row_to_memory) + .map_err(|e| Error::db("database operation", e))?; + for row in rows { + let m = row.map_err(|e| Error::db("database operation", e))?; + memories.push(m); + } + } + } + Ok(memories) + } + /// Count total memories, optionally filtered by namespace. /// Count ACTIVE (non-deprecated) memories, optionally filtered by namespace. /// diff --git a/crates/uteke-core/src/memory/rooms.rs b/crates/uteke-core/src/memory/rooms.rs index a5d5adb6..1a3c1f89 100644 --- a/crates/uteke-core/src/memory/rooms.rs +++ b/crates/uteke-core/src/memory/rooms.rs @@ -755,8 +755,19 @@ impl super::Store { Ok(Some(enriched)) } - /// Delete a room and all its memory links (CASCADE). - pub fn delete_room(&self, room_id: &str) -> Result<(), Error> { + /// Delete a room (unlink-only). + /// + /// Removes the room row; the `room_memories` and `room_documents` links + /// are removed by `ON DELETE CASCADE`. The linked memories and documents + /// themselves are NOT deleted — they remain in their namespaces, now + /// orphaned from any room. + /// + /// Returns the number of memory links that were removed. + pub fn delete_room(&self, room_id: &str) -> Result { + let memory_links = self + .get_room_memory_ids(room_id, None) + .map_err(|e| Error::db_msg(format!("failed to count room memories: {e}")))? + .len(); let rows = self .conn .execute("DELETE FROM rooms WHERE id = ?1", params![room_id]) @@ -764,7 +775,7 @@ impl super::Store { if rows == 0 { return Err(Error::db_msg(format!("Room not found: {room_id}"))); } - Ok(()) + Ok(memory_links) } /// Generate a structured document from room memories, grouped by memory_type. @@ -1160,7 +1171,8 @@ mod tests { fn delete_room_success() { let store = Store::open(":memory:").unwrap(); store.create_room("del-me", None, "default").unwrap(); - store.delete_room("del-me").unwrap(); + let unlinked = store.delete_room("del-me").unwrap(); + assert_eq!(unlinked, 0); assert!(store.get_room("del-me").unwrap().is_none()); } @@ -1186,7 +1198,8 @@ mod tests { let ids = store.get_room_memory_ids("cascade-room", None).unwrap(); assert_eq!(ids.len(), 1); - store.delete_room("cascade-room").unwrap(); + let unlinked = store.delete_room("cascade-room").unwrap(); + assert_eq!(unlinked, 1); // Memory itself survives — only the room_memories link is cascade-deleted assert!(store.get_by_id("mem-1").unwrap().is_some()); // Room is gone, so recall_room should return empty (room doesn't exist) diff --git a/crates/uteke-core/src/memory/schema.rs b/crates/uteke-core/src/memory/schema.rs index 0166933e..4e9dd367 100644 --- a/crates/uteke-core/src/memory/schema.rs +++ b/crates/uteke-core/src/memory/schema.rs @@ -418,6 +418,8 @@ impl super::Store { 16 => self.migrate_v15_to_v16()?, // v17: deprecated_at column — time-travel deprecation predicate (#1086) 17 => self.migrate_v16_to_v17()?, + // v18: provenance chain — source_hash, actor, evidence_json (#1172) + 18 => self.migrate_v17_to_v18()?, _ => { // No-op for future versions. } @@ -604,6 +606,7 @@ impl super::Store { "memory_feedback", "memory_doc_refs", "doc_mem_refs", + "timeline_events", ]; if !ALLOWED_TABLES.contains(&table) { return false; @@ -1129,4 +1132,52 @@ impl super::Store { tracing::info!("Migration v16 to v17 complete: deprecated_at column added"); Ok(()) } + + /// v18: Provenance chain fields (#1172 Fase 1). + /// + /// Additive, zero data loss: + /// - `memories.source_hash` — SHA-256 of the content at write time + /// (tamper-evidence for audits). + /// - `timeline_events.actor` — who performed the event (agent id, "user", + /// "system"). + /// - `timeline_events.evidence_json` — JSON array of related memory IDs / + /// scores supporting the event (e.g. contradiction resolution evidence). + fn migrate_v17_to_v18(&self) -> Result<(), Error> { + tracing::info!("Applying schema migration v17 to v18: provenance chain (#1172)"); + + if !self.column_exists("source_hash") { + self.conn + .execute_batch("ALTER TABLE memories ADD COLUMN source_hash TEXT;") + .map_err(|e| Error::db("schema migration v17 to v18: source_hash", e))?; + } + // Guard for stores missing the v9 table entirely (defensive — a + // stamped v17 store should have it, but test fixtures / repaired + // databases may not). Fresh shape includes the new columns. + self.conn + .execute_batch( + "CREATE TABLE IF NOT EXISTS timeline_events ( + id INTEGER PRIMARY KEY AUTOINCREMENT, + memory_id TEXT NOT NULL REFERENCES memories(id) ON DELETE CASCADE, + event_type TEXT NOT NULL, + event_data TEXT, + created_at TEXT NOT NULL, + actor TEXT, + evidence_json TEXT + );", + ) + .map_err(|e| Error::db("schema migration v17 to v18: timeline_events", e))?; + if !self.column_exists_in("timeline_events", "actor") { + self.conn + .execute_batch("ALTER TABLE timeline_events ADD COLUMN actor TEXT;") + .map_err(|e| Error::db("schema migration v17 to v18: actor", e))?; + } + if !self.column_exists_in("timeline_events", "evidence_json") { + self.conn + .execute_batch("ALTER TABLE timeline_events ADD COLUMN evidence_json TEXT;") + .map_err(|e| Error::db("schema migration v17 to v18: evidence_json", e))?; + } + + tracing::info!("Migration v17 to v18 complete: provenance chain columns added"); + Ok(()) + } } diff --git a/crates/uteke-core/src/memory/store.rs b/crates/uteke-core/src/memory/store.rs index ade512cb..dd138a6f 100644 --- a/crates/uteke-core/src/memory/store.rs +++ b/crates/uteke-core/src/memory/store.rs @@ -31,7 +31,8 @@ CREATE TABLE IF NOT EXISTS memories ( slug TEXT, source TEXT, source_type TEXT NOT NULL DEFAULT 'user', - author_type TEXT NOT NULL DEFAULT 'agent' + author_type TEXT NOT NULL DEFAULT 'agent', + source_hash TEXT ); CREATE INDEX IF NOT EXISTS idx_memories_tags ON memories(tags); CREATE INDEX IF NOT EXISTS idx_memories_created ON memories(created_at); @@ -83,7 +84,9 @@ CREATE TABLE IF NOT EXISTS timeline_events ( memory_id TEXT NOT NULL REFERENCES memories(id) ON DELETE CASCADE, event_type TEXT NOT NULL, event_data TEXT, - created_at TEXT NOT NULL + created_at TEXT NOT NULL, + actor TEXT, + evidence_json TEXT ); CREATE INDEX IF NOT EXISTS idx_timeline_memory ON timeline_events(memory_id); CREATE INDEX IF NOT EXISTS idx_timeline_type ON timeline_events(event_type); @@ -176,7 +179,7 @@ pub(super) const SCHEMA_INDEXES: &[&str] = &[ ]; /// Current schema version. Increment when adding migrations. -pub(crate) const CURRENT_SCHEMA_VERSION: i32 = 17; +pub(crate) const CURRENT_SCHEMA_VERSION: i32 = 18; /// Persistent SQLite store for memories. pub struct Store { @@ -272,6 +275,34 @@ impl Store { Ok(rows > 0) } + /// Set the provenance content hash for a memory (#1172 Fase 1). + /// + /// `None` clears the hash (used by repair/backfill tooling). Returns + /// `false` when the memory does not exist. + pub fn set_source_hash(&self, id: &str, source_hash: Option<&str>) -> Result { + let rows = self + .conn + .execute( + "UPDATE memories SET source_hash = ?1 WHERE id = ?2", + rusqlite::params![source_hash, id], + ) + .map_err(|e| Error::db("set source hash", e))?; + Ok(rows > 0) + } + + /// Read the provenance content hash for a memory (#1172 Fase 1). + pub fn get_source_hash(&self, id: &str) -> Result, Error> { + use rusqlite::OptionalExtension; + self.conn + .query_row( + "SELECT source_hash FROM memories WHERE id = ?1", + rusqlite::params![id], + |row| row.get(0), + ) + .optional() + .map_err(|e| Error::db("get source hash", e)) + } + /// Set author type on a memory (#1083): "human" or "agent". /// Returns false if the memory does not exist. pub fn set_author_type(&self, id: &str, author_type: &str) -> Result { @@ -345,6 +376,25 @@ impl Store { } Ok(updated) } + + /// Infer embedding dimensions from any persisted embedding (#1166). + /// + /// Returns None when the store has no embeddings at all (fresh store). + /// Used when opening without an embedder backend on builds compiled + /// without the `onnx` feature: the vector index needs valid dims, and + /// existing data is the most truthful source. + pub fn infer_embedding_dims(&self) -> Option { + self.conn + .query_row( + "SELECT length(embedding) / 4 FROM memories \ + WHERE embedding IS NOT NULL AND length(embedding) > 0 \ + LIMIT 1", + [], + |row| row.get::<_, i64>(0), + ) + .ok() + .map(|n| n as usize) + } } /// Serialize an embedding vector to a byte blob (little-endian f32). @@ -683,7 +733,7 @@ mod tests { .conn .query_row("SELECT MAX(version) FROM schema_version", [], |r| r.get(0)) .unwrap(); - assert_eq!(version, 17, "schema should be upgraded to v17"); + assert_eq!(version, 18, "schema should be upgraded to current (v18)"); // Legacy row backfilled to 'agent'. let at: String = store diff --git a/crates/uteke-core/src/memory/tags.rs b/crates/uteke-core/src/memory/tags.rs index 7fb36445..918731f4 100644 --- a/crates/uteke-core/src/memory/tags.rs +++ b/crates/uteke-core/src/memory/tags.rs @@ -382,4 +382,139 @@ impl super::Store { } Ok(result) } + + /// List namespaces with counts split by lifecycle state (#1181). + /// + /// Returns `[(namespace, active, deprecated)]`. Deprecated-only "ghost" + /// namespaces currently look identical to active ones in listings; + /// splitting the counts lets clients show honest breakdowns. + pub fn list_namespaces_with_lifecycle_counts( + &self, + ) -> Result, Error> { + let mut stmt = self + .conn + .prepare( + "SELECT namespace, \ + SUM(CASE WHEN deprecated = 0 THEN 1 ELSE 0 END), \ + SUM(CASE WHEN deprecated = 1 THEN 1 ELSE 0 END) \ + FROM memories \ + GROUP BY namespace \ + ORDER BY namespace", + ) + .map_err(|e| Error::db("database operation", e))?; + + let rows = stmt + .query_map([], |row: &rusqlite::Row| { + Ok(( + row.get::<_, String>(0)?, + row.get::<_, i64>(1)?.max(0) as usize, + row.get::<_, i64>(2)?.max(0) as usize, + )) + }) + .map_err(|e| Error::db("database operation", e))?; + + let mut result = Vec::new(); + for row in rows { + result.push(row.map_err(|e| Error::db("database operation", e))?); + } + Ok(result) + } + + /// Count a namespace's memories split by lifecycle state (#1181). + /// + /// Returns `(active, deprecated, total)` — `total` includes deprecated. + pub fn namespace_lifecycle_counts(&self, name: &str) -> Result<(usize, usize, usize), Error> { + self.conn + .query_row( + "SELECT \ + COALESCE(SUM(CASE WHEN deprecated = 0 THEN 1 ELSE 0 END), 0), \ + COALESCE(SUM(CASE WHEN deprecated = 1 THEN 1 ELSE 0 END), 0), \ + COUNT(*) \ + FROM memories WHERE namespace = ?1", + params![name], + |row: &rusqlite::Row| { + Ok(( + row.get::<_, i64>(0)?.max(0) as usize, + row.get::<_, i64>(1)?.max(0) as usize, + row.get::<_, i64>(2)?.max(0) as usize, + )) + }, + ) + .map_err(|e| Error::db("namespace lifecycle counts", e)) + } + + /// Move a single memory to another namespace (#1181). + /// + /// Returns `false` when the memory does not exist. Embeddings are + /// content-based, so no re-embed is needed — namespace is a plain column. + pub fn move_memory_namespace( + &self, + id: &str, + namespace: &str, + now: &str, + ) -> Result { + let rows = self + .conn + .execute( + "UPDATE memories SET namespace = ?1, updated_at = ?2 WHERE id = ?3", + params![namespace, now, id], + ) + .map_err(|e| Error::db("move memory namespace", e))?; + Ok(rows > 0) + } + + /// Atomically rename a namespace (merge when `to` already exists) (#1181). + /// + /// Returns the number of memories moved. The old name vanishes naturally: + /// namespaces are a derived view over the `memories.namespace` column. + pub fn rename_namespace(&self, from: &str, to: &str, now: &str) -> Result { + let tx = self + .conn + .unchecked_transaction() + .map_err(|e| Error::db("begin namespace rename", e))?; + let rows = tx + .execute( + "UPDATE memories SET namespace = ?1, updated_at = ?2 WHERE namespace = ?3", + params![to, now, from], + ) + .map_err(|e| Error::db("rename namespace", e))?; + tx.commit() + .map_err(|e| Error::db("commit namespace rename", e))?; + Ok(rows) + } + + /// All memory IDs in a namespace, including deprecated (#1181). + pub fn namespace_ids(&self, name: &str) -> Result, Error> { + let mut stmt = self + .conn + .prepare("SELECT id FROM memories WHERE namespace = ?1") + .map_err(|e| Error::db("prepare namespace ids", e))?; + let rows = stmt + .query_map(params![name], |row: &rusqlite::Row| row.get::<_, String>(0)) + .map_err(|e| Error::db("namespace ids", e))?; + let mut ids = Vec::new(); + for row in rows { + ids.push(row.map_err(|e| Error::db("namespace ids", e))?); + } + Ok(ids) + } + + /// Soft-delete (deprecate) every active memory in a namespace (#1181). + /// + /// Never hard-deletes — deprecated memories stay restorable via + /// `promote()`, consistent with the lifecycle philosophy (#929). + /// Returns the number of newly deprecated rows. + pub fn deprecate_by_namespace(&self, name: &str, reason: &str) -> Result { + let now = chrono::Utc::now().to_rfc3339(); + let rows = self + .conn + .execute( + "UPDATE memories SET deprecated = 1, valid_until = ?1, deprecate_reason = ?2, \ + updated_at = ?1, deprecated_at = ?1 \ + WHERE namespace = ?3 AND deprecated = 0", + params![now, reason, name], + ) + .map_err(|e| Error::db("deprecate namespace", e))?; + Ok(rows) + } } diff --git a/crates/uteke-core/src/memory/types.rs b/crates/uteke-core/src/memory/types.rs index 4efa712a..aa7f08f8 100644 --- a/crates/uteke-core/src/memory/types.rs +++ b/crates/uteke-core/src/memory/types.rs @@ -307,6 +307,35 @@ pub struct BulkDeleteResult { pub ids: Vec, } +/// Result of a namespace delete with an explicit strategy (#1181). +#[derive(Debug, Clone, Serialize, Deserialize)] +pub struct NamespaceDeleteResult { + /// The deleted namespace name. + pub name: String, + /// Strategy that was applied. + pub strategy: String, + /// Memories affected by the strategy. + pub affected: usize, + /// Memories moved to `target` (only for the `merge` strategy). + #[serde(skip_serializing_if = "Option::is_none")] + pub target: Option, + /// True when the namespace is now empty and its name disappears from listings. + pub empty: bool, +} + +/// Result of a namespace rename/merge (#1181). +#[derive(Debug, Clone, Serialize, Deserialize)] +pub struct NamespaceRenameResult { + /// The original namespace name. + pub from: String, + /// The new namespace name. + pub to: String, + /// Number of memories moved. + pub moved: usize, + /// True when `to` already existed (this call was a merge). + pub target_existed: bool, +} + /// Lightweight export format — no embedding vector (re-embedded on import). #[derive(Debug, Clone, Serialize, Deserialize)] pub struct ExportEntry { diff --git a/crates/uteke-core/src/memory/vector.rs b/crates/uteke-core/src/memory/vector.rs index 5301d216..898b29a2 100644 --- a/crates/uteke-core/src/memory/vector.rs +++ b/crates/uteke-core/src/memory/vector.rs @@ -1,15 +1,27 @@ //! Persistent vector index with pluggable backends. //! -//! Two mutually exclusive backends, selected at compile time via cargo features: +//! Backends are selected at RUNTIME (#1168) while remaining compile-time +//! slim-buildable: //! -//! - `usearch` (default): HNSW with disk persistence via C++ FFI. -//! - `vecq`: training-free 4-bit vector quantization, pure Rust, no C++ -//! toolchain (for mobile/FFI builds, #1098). +//! - `usearch`: HNSW with disk persistence via C++ FFI. +//! - `vecq`: training-free 4-bit + residual vector quantization, pure Rust, +//! no C++ toolchain (originally for mobile/FFI builds, #1098). +//! +//! A build may compile in one or both engines (cargo features `usearch` / +//! `vecq`). Which engine a `VectorIndex` uses is decided when it is created: +//! `UTEKE_VECTOR_BACKEND` env → explicit constructor argument (from +//! `uteke.toml [vector] backend`) → compiled-in default. At least one engine +//! feature must be enabled. //! //! The public API (`new`, `load_or_create`, `insert`, `remove`, `search`, //! `build`, `save`, `len`, `dims`, ...) is identical for both backends — //! callers never branch on the backend. //! +//! Index files are per-engine: `uteke_index.usearch` vs `uteke_index.vecq`. +//! Switching engines leaves the old file untouched; the new engine finds no +//! index and the caller (lib.rs `finish_open_full`) rebuilds from SQLite, +//! which remains the source of truth. +//! //! Cross-process safety (#543): Each VectorIndex acquires an exclusive file //! lock (via fs2) on the index file during construction. The lock is held //! until the VectorIndex is dropped, serializing concurrent CLI invocations @@ -23,17 +35,14 @@ //! via Rust std::fs; load reads via Rust std::fs then deserializes from buffer. //! //! vecq format note: vecq has no incremental delete — removed rows are -//! tombstoned (tracked via the `dead` bitmap) and filtered out of search +//! tombstoned (the key vanishes from the key map) and filtered out of search //! results. Tombstones are derived from the key-mapping sidecar on load, so //! no extra on-disk state is needed. -#[cfg(all(feature = "usearch", feature = "vecq"))] +#[cfg(not(any(feature = "usearch", feature = "vecq")))] compile_error!( - "features `usearch` and `vecq` are mutually exclusive vector index backends; \ - enable exactly one (default build uses `usearch`)" + "uteke-core requires a vector index backend; enable `usearch` (default) and/or `vecq`" ); -#[cfg(not(any(feature = "usearch", feature = "vecq")))] -compile_error!("uteke-core requires a vector index backend; enable `usearch` (default) or `vecq`"); use crate::Error; use fs2::FileExt; @@ -46,11 +55,12 @@ use usearch::{Index, IndexOptions, MetricKind, ScalarKind}; #[cfg(feature = "vecq")] use vecq_core::VecqIndex; -/// Extension for the on-disk index file. +/// Extension for the on-disk index file of the compiled-in DEFAULT backend. +/// +/// Prefer [`index_ext_for`] when a specific backend is selected at runtime. #[cfg(feature = "usearch")] pub const INDEX_EXT: &str = "usearch"; -/// Extension for the on-disk index file (vecq backend, #1098). -#[cfg(feature = "vecq")] +#[cfg(not(feature = "usearch"))] pub const INDEX_EXT: &str = "vecq"; /// Default dimensions for EmbeddingGemma Q4 (768d). @@ -61,11 +71,296 @@ const DEFAULT_DIMS: usize = 768; #[cfg(feature = "vecq")] const VECQ_SEED: u64 = 0x7574_656b; // "utek" +/// Which vector engine a `VectorIndex` uses (#1168). +#[derive(Clone, Copy, Debug, PartialEq, Eq)] +pub enum VectorBackend { + Usearch, + Vecq, +} + +impl VectorBackend { + /// Parse a user-facing backend name (env var / toml value). + pub fn parse(s: &str) -> Option { + match s.trim().to_ascii_lowercase().as_str() { + "usearch" => Some(VectorBackend::Usearch), + "vecq" => Some(VectorBackend::Vecq), + _ => None, + } + } + + /// The compiled-in default engine: usearch when available (long-standing + /// desktop default, best query latency), else vecq (slim builds). + pub fn default_backend() -> Self { + default_backend() + } + + /// Resolve a preferred backend against the engines compiled in. + /// + /// Returns `(resolved, honored)` where `honored` is false when the + /// preference had to fall back (engine not compiled in), or when the + /// preference was `None`. + pub fn resolve(preferred: Option) -> (Self, bool) { + let default = default_backend(); + match preferred { + None => (default, false), + Some(p) if p.is_compiled_in() => (p, true), + Some(_p) => (default, false), + } + } + + /// Whether this engine's cargo feature is compiled into the binary. + pub fn is_compiled_in(self) -> bool { + match self { + #[cfg(feature = "usearch")] + VectorBackend::Usearch => true, + #[cfg(not(feature = "usearch"))] + VectorBackend::Usearch => false, + #[cfg(feature = "vecq")] + VectorBackend::Vecq => true, + #[cfg(not(feature = "vecq"))] + VectorBackend::Vecq => false, + } + } +} + +/// Compiled-in default engine (see [`VectorBackend::default_backend`]). +pub fn default_backend() -> VectorBackend { + #[cfg(feature = "usearch")] + { + VectorBackend::Usearch + } + #[cfg(not(feature = "usearch"))] + { + VectorBackend::Vecq + } +} + +/// On-disk index file extension for a backend. +pub fn index_ext_for(backend: VectorBackend) -> &'static str { + match backend { + VectorBackend::Usearch => "usearch", + VectorBackend::Vecq => "vecq", + } +} + +/// The concrete engine instance — one variant per compiled-in backend. +#[allow(clippy::large_enum_variant)] +enum Engine { + #[cfg(feature = "usearch")] + Usearch(Box), + #[cfg(feature = "vecq")] + Vecq(VecqIndex), +} + +impl Engine { + /// Create an empty engine of `backend` for `dims`-dimensional vectors. + fn create(backend: VectorBackend, dims: usize) -> Result { + match backend { + #[cfg(feature = "usearch")] + VectorBackend::Usearch => { + let options = IndexOptions { + dimensions: dims, + metric: MetricKind::Cos, + quantization: ScalarKind::F32, + ..Default::default() + }; + let index = Index::new(&options).map_err(|e| { + Error::embed_msg(format!( + "Failed to create usearch index (dims={dims}): {e}. This is likely an out-of-memory condition." + )) + })?; + Ok(Engine::Usearch(Box::new(index))) + } + #[cfg(not(feature = "usearch"))] + VectorBackend::Usearch => Err(Error::validation( + "usearch engine requested but not compiled in", + )), + #[cfg(feature = "vecq")] + VectorBackend::Vecq => Ok(Engine::Vecq(VecqIndex::with_residual(dims, VECQ_SEED))), + #[cfg(not(feature = "vecq"))] + VectorBackend::Vecq => Err(Error::validation( + "vecq engine requested but not compiled in", + )), + } + } + + /// Restore an engine from a serialized buffer. The buffer must have been + /// written by the SAME engine (`Engine::to_bytes`). + fn from_bytes(backend: VectorBackend, buffer: &[u8]) -> Result { + match backend { + #[cfg(feature = "usearch")] + VectorBackend::Usearch => { + let index = Index::restore_from_buffer(buffer) + .map_err(|e| Error::embed("load vector index", e))?; + Ok(Engine::Usearch(Box::new(index))) + } + #[cfg(not(feature = "usearch"))] + VectorBackend::Usearch => Err(Error::validation( + "usearch engine requested but not compiled in", + )), + #[cfg(feature = "vecq")] + VectorBackend::Vecq => { + let index = VecqIndex::from_bytes(buffer) + .map_err(|e| Error::embed("load vector index (vecq)", e))?; + Ok(Engine::Vecq(index)) + } + #[cfg(not(feature = "vecq"))] + VectorBackend::Vecq => Err(Error::validation( + "vecq engine requested but not compiled in", + )), + } + } + + /// Serialize to an in-memory buffer (see save() notes on buffer-based I/O). + fn to_bytes(&self) -> Result, Error> { + match self { + #[cfg(feature = "usearch")] + Engine::Usearch(index) => { + let buf_len = index.serialized_length(); + let mut buffer = vec![0u8; buf_len]; + index + .save_to_buffer(&mut buffer) + .map_err(|e| Error::embed("save vector index to buffer", e))?; + Ok(buffer) + } + #[cfg(feature = "vecq")] + Engine::Vecq(index) => Ok(index.to_bytes()), + } + } + + /// Number of physical rows/entries in the engine (including vecq + /// tombstoned rows; live count is the key map's length). + fn len(&self) -> usize { + match self { + #[cfg(feature = "usearch")] + Engine::Usearch(index) => index.size(), + #[cfg(feature = "vecq")] + Engine::Vecq(index) => index.len(), + } + } + + /// Capacity (diagnostics). + fn capacity(&self) -> usize { + match self { + #[cfg(feature = "usearch")] + Engine::Usearch(index) => index.capacity(), + #[cfg(feature = "vecq")] + Engine::Vecq(index) => index.len(), + } + } + + /// Embedding dimensionality. + fn dims(&self) -> usize { + match self { + #[cfg(feature = "usearch")] + Engine::Usearch(index) => index.dimensions(), + #[cfg(feature = "vecq")] + Engine::Vecq(index) => index.dim(), + } + } + + /// Pre-reserve capacity for a bulk build (usearch only; vecq grows + /// organically). + fn reserve(&mut self, #[allow(unused_variables)] additional: usize) { + match self { + #[cfg(feature = "usearch")] + Engine::Usearch(index) => { + if let Err(e) = index.reserve(additional) { + tracing::error!("Failed to reserve usearch capacity: {e}"); + } + } + #[cfg(feature = "vecq")] + Engine::Vecq(_) => { /* vecq: no-op */ } + } + } + + /// Auto-grow capacity when full (usearch only). + fn ensure_capacity(&mut self) -> Result<(), Error> { + match self { + #[cfg(feature = "usearch")] + Engine::Usearch(index) => { + if index.size() >= index.capacity() { + // Auto-reserve using geometric growth to amortize reallocation cost. + // Growth strategy: max(current * 2, current + 4096, 1024). + let current = index.capacity(); + let new_cap = (current * 2).max(current + 4096).max(1024); + index.reserve(new_cap).map_err(|e| { + Error::embed_msg(format!("Failed to reserve usearch capacity: {e}")) + })?; + } + Ok(()) + } + #[cfg(feature = "vecq")] + Engine::Vecq(_) => Ok(()), + } + } + + /// Insert a vector under `key`. + /// + /// usearch: keyed insert. vecq: append-only — the row index MUST equal + /// `key` (the caller maintains the row == key invariant by deriving keys + /// from `Engine::len()`), the key argument is ignored. + fn add(&mut self, #[allow(unused_variables)] key: u64, embedding: &[f32]) -> Result<(), Error> { + match self { + #[cfg(feature = "usearch")] + Engine::Usearch(index) => index + .add(key, embedding) + .map_err(|e| Error::embed_msg(format!("Failed to insert into usearch index: {e}"))), + #[cfg(feature = "vecq")] + Engine::Vecq(index) => { + index.add(embedding); + Ok(()) + } + } + } + + /// Remove a key (usearch). vecq has no incremental delete — the caller + /// tombstones via the key map instead. + fn remove(&mut self, #[allow(unused_variables)] key: u64) { + match self { + #[cfg(feature = "usearch")] + Engine::Usearch(index) => { + if let Err(e) = index.remove(key) { + tracing::error!("Failed to remove from usearch index: {e}"); + } + } + #[cfg(feature = "vecq")] + Engine::Vecq(_) => { /* tombstone via key map (see remove()) */ } + } + } + + /// Top-`k` search. Returns `(key, cosine_distance)` pairs sorted by + /// distance ascending, engine-agnostic (vecq rows ARE keys). + fn search(&self, query: &[f32], k: usize) -> Vec<(u64, f32)> { + match self { + #[cfg(feature = "usearch")] + Engine::Usearch(index) => match index.search(query, k) { + Ok(r) => r + .keys + .iter() + .copied() + .zip(r.distances.iter().copied()) + .collect(), + Err(e) => { + tracing::error!("usearch search failed: {e}"); + Vec::new() + } + }, + #[cfg(feature = "vecq")] + Engine::Vecq(index) => index + .search(query, k) + .into_iter() + .map(|(row, sim)| (row as u64, 1.0 - sim)) + .collect(), + } + } +} + /// Persistent vector index. /// /// - **Startup**: loads from disk (~5ms), no rebuild needed /// - **Insert**: incremental, no rebuild -/// - **Delete**: incremental, no rebuild +/// - **Delete**: incremental, no rebuild (vecq: tombstone) /// - **Save**: persists to disk after mutations /// /// **Cross-process safety (#543):** An exclusive file lock on the index @@ -73,10 +368,11 @@ const VECQ_SEED: u64 = 0x7574_656b; // "utek" /// held for the lifetime of the VectorIndex. In-process thread safety uses /// `RwLock` in `Uteke`. pub struct VectorIndex { - #[cfg(feature = "usearch")] - index: Index, - #[cfg(feature = "vecq")] - index: VecqIndex, + /// Active engine (runtime-selected, #1168). + engine: Engine, + /// Which backend `engine` is (mirrors the enum variant; kept separately + /// for cheap comparisons without matching). + backend: VectorBackend, /// Maps integer key (u64) → memory UUID string. key_to_id: HashMap, /// Maps memory UUID → integer key. @@ -93,11 +389,22 @@ pub struct VectorIndex { } impl VectorIndex { - /// Create a new empty vector index. + /// The engine this index runs on (#1168). + pub fn backend(&self) -> VectorBackend { + self.backend + } + + /// Create a new empty vector index using the compiled-in default engine. pub fn new(dims: usize) -> Result { - let index = Self::create_index(dims)?; + Self::with_backend(default_backend(), dims) + } + + /// Create a new empty vector index for an explicit engine (#1168). + pub fn with_backend(backend: VectorBackend, dims: usize) -> Result { + let engine = Engine::create(backend, dims)?; Ok(Self { - index, + engine, + backend, key_to_id: HashMap::new(), id_to_key: HashMap::new(), next_key: 0, @@ -108,7 +415,8 @@ impl VectorIndex { } /// Load index from disk, or create empty if file doesn't exist. - /// `path` is the path to the index file. + /// `path` is the path to the index file. The engine is inferred from the + /// file extension (`uteke_index.usearch` / `uteke_index.vecq`). /// /// Acquires an **exclusive file lock** on the index file to prevent /// cross-process race conditions (e.g., `xargs -P5 uteke remember`). @@ -119,6 +427,22 @@ impl VectorIndex { /// identical — `save_to_buffer` and `restore_from_buffer` produce/consume /// the same byte stream as the native file-based methods. pub fn load_or_create(path: &Path, dims: usize) -> Result { + // The extension determines which engine reads the file (#1112 kept + // per-engine extensions for exactly this purpose). + let backend = match path.extension().and_then(|e| e.to_str()) { + Some("vecq") => VectorBackend::Vecq, + _ => VectorBackend::Usearch, + }; + Self::load_or_create_for(path, dims, backend) + } + + /// [`load_or_create`] with an explicit engine (#1168). Use when the path + /// may not carry a backend-specific extension. + pub fn load_or_create_for( + path: &Path, + dims: usize, + backend: VectorBackend, + ) -> Result { // Atomically create the file if it doesn't exist (avoids TOCTOU race // where another process creates the file between our exists() and write()). // O_CREAT | O_EXCL ensures only one writer wins; failure is harmless. @@ -133,9 +457,9 @@ impl VectorIndex { .len() == 0 { - Self::new(dims)? + Self::with_backend(backend, dims)? } else { - Self::load_from_file(&mut lock_file, path)? + Self::load_from_file_for(&mut lock_file, path, backend)? }; idx.path = Some(path.to_path_buf()); idx._lock_file = Some(lock_file); @@ -148,6 +472,19 @@ impl VectorIndex { /// file is locked exclusively by the same process (#732), since we read /// directly from the locked handle instead of opening a second one. pub fn load_from_file(file: &mut File, path: &Path) -> Result { + let backend = match path.extension().and_then(|e| e.to_str()) { + Some("vecq") => VectorBackend::Vecq, + _ => VectorBackend::Usearch, + }; + Self::load_from_file_for(file, path, backend) + } + + /// [`load_from_file`] with an explicit engine (#1168). + pub fn load_from_file_for( + file: &mut File, + path: &Path, + backend: VectorBackend, + ) -> Result { use std::io::{Read, Seek, SeekFrom}; file.seek(SeekFrom::Start(0)) @@ -157,15 +494,17 @@ impl VectorIndex { file.read_to_end(&mut buffer) .map_err(|e| Error::embed("read index file from locked handle", e))?; - #[cfg(feature = "usearch")] - let index = Index::restore_from_buffer(&buffer) - .map_err(|e| Error::embed("load vector index", e))?; - #[cfg(feature = "usearch")] - let _ = index.size(); - - #[cfg(feature = "vecq")] - let index = VecqIndex::from_bytes(&buffer) - .map_err(|e| Error::embed("load vector index (vecq)", e))?; + let engine = Engine::from_bytes(backend, &buffer)?; + // Legacy usearch-only builds: touch size() to validate the restored + // handle (kept from the pre-#1168 load path). On builds with both + // engines the validation already happened inside from_bytes. + #[cfg(all(feature = "usearch", not(feature = "vecq")))] + let engine = match engine { + Engine::Usearch(idx) => { + let _ = idx.size(); + Engine::Usearch(idx) + } + }; // Rebuild key mappings from the sidecar file let mut key_to_id = HashMap::new(); @@ -196,7 +535,8 @@ impl VectorIndex { } Ok(Self { - index, + engine, + backend, key_to_id, id_to_key, next_key, @@ -226,7 +566,7 @@ impl VectorIndex { /// This bypasses usearch's C++ `fopen("wb")` file I/O which has known /// issues on Windows: /// - `fopen` fails silently on paths > 260 chars (MAX_PATH) - /// - `fopen("wb")` exclusive access conflicts with `fs2` exclusive lock + /// - `fopen("wb")` exclusive access conflicts with fs2 exclusive lock /// - Windows Defender can intercept `fwrite` calls /// /// The in-memory buffer approach is safe because: @@ -235,20 +575,10 @@ impl VectorIndex { pub fn save(&mut self) -> Result<(), Error> { if let Some(ref path) = self.path { // Serialize index to in-memory buffer, bypassing C++ file I/O (#647) - #[cfg(feature = "usearch")] - let buffer: Vec = { - let buf_len = self.index.serialized_length(); - let mut buffer = vec![0u8; buf_len]; - self.index - .save_to_buffer(&mut buffer) - .map_err(|e| Error::embed("save vector index to buffer", e))?; - buffer - }; - #[cfg(feature = "vecq")] - let buffer: Vec = self.index.to_bytes(); + let buffer = self.engine.to_bytes()?; // Write buffer to disk via atomic write (temp file + rename) - let tmp_path = path.with_extension(format!("{INDEX_EXT}.tmp")); + let tmp_path = path.with_extension(format!("{}.tmp", index_ext_for(self.backend))); std::fs::write(&tmp_path, &buffer) .map_err(|e| Error::embed("write temp index file", e))?; @@ -322,13 +652,13 @@ impl VectorIndex { /// Build the index from a list of (id, embedding) pairs. /// Used for migration from old HNSW or full rebuild. pub fn build(&mut self, items: &[(String, Vec)]) -> Result<(), Error> { - // Reset + // Reset (same engine, fresh instance) let dims = if items.is_empty() { DEFAULT_DIMS } else { items[0].1.len() }; - self.index = Self::create_index(dims)?; + self.engine = Engine::create(self.backend, dims)?; self.key_to_id.clear(); self.id_to_key.clear(); self.next_key = 0; @@ -343,10 +673,7 @@ impl VectorIndex { ))); } } - #[cfg(feature = "usearch")] - if let Err(e) = self.index.reserve(items.len()) { - tracing::error!("Failed to reserve usearch capacity: {e}"); - } + self.engine.reserve(items.len()); } for (id, embedding) in items { @@ -361,12 +688,11 @@ impl VectorIndex { pub fn insert(&mut self, id: &str, embedding: &[f32]) -> Result<(), Error> { // Validate dimensions up front, before any map mutation, so error // paths leave the key maps consistent with the physical index. - #[cfg(feature = "vecq")] - if embedding.len() != self.index.dim() { + if embedding.len() != self.engine.dims() { return Err(Error::validation(format!( "embedding dimension mismatch: got {}, expected {}", embedding.len(), - self.index.dim() + self.engine.dims() ))); } @@ -374,67 +700,43 @@ impl VectorIndex { if let Some(old_key) = self.id_to_key.get(id) { let old_key = *old_key; self.key_to_id.remove(&old_key); - #[cfg(feature = "usearch")] - self.index.remove(old_key).map_err(|e| { - Error::embed_msg(format!( - "Failed to remove old entry for duplicate ID {id}: {e}" - )) - })?; + self.engine.remove(old_key); // vecq has no incremental delete — the dead row is filtered out of // search via the key map (key_to_id no longer contains old_key). } - #[cfg(feature = "vecq")] - { + let key = if self.backend == VectorBackend::Vecq { // vecq rows are append-only: the new entry lands at physical row - // `index.len()`, and search maps rows back via key_to_id — so the + // `len()`, and search maps rows back via key_to_id — so the // key MUST equal that row exactly (no gaps). // // Crash window: save() writes the index file and the `.keys` // sidecar as two separate atomic writes. If the process dies after // the sidecar but before the index, the reloaded sidecar can hold - // keys ≥ index.len() ("phantom" keys pointing at rows that were + // keys ≥ len() ("phantom" keys pointing at rows that were // never written). Overwriting such a phantom key here is correct: // its row never existed, so nothing live can be shadowed. - let key = self.index.len() as u64; + let key = self.engine.len() as u64; if let Some(phantom_id) = self.key_to_id.remove(&key) { tracing::warn!( "overwriting phantom key {key} (id '{phantom_id}' had no row in the vecq index — likely a crash between sidecar and index writes)" ); self.id_to_key.remove(&phantom_id); } - self.next_key = key.saturating_add(1); - - self.key_to_id.insert(key, id.to_string()); - self.id_to_key.insert(id.to_string(), key); - } - #[cfg(feature = "usearch")] - { + key + } else { let key = self.next_key; self.next_key = self.next_key.saturating_add(1); + // Auto-grow usearch capacity when full. + self.engine.ensure_capacity()?; + key + }; - self.key_to_id.insert(key, id.to_string()); - self.id_to_key.insert(id.to_string(), key); - - // Auto-reserve if at capacity using geometric growth to amortize reallocation cost. - // Growth strategy: max(current * 2, current + 4096, 1024). - // Doubling amortizes to O(1) per insertion; +4096 floor avoids tiny allocs at small scale. - if self.index.size() >= self.index.capacity() { - let current = self.index.capacity(); - let new_cap = (current * 2).max(current + 4096).max(1024); - self.index.reserve(new_cap).map_err(|e| { - Error::embed_msg(format!("Failed to reserve usearch capacity: {e}")) - })?; - } + self.key_to_id.insert(key, id.to_string()); + self.id_to_key.insert(id.to_string(), key); - self.index.add(key, embedding).map_err(|e| { - Error::embed_msg(format!("Failed to insert into usearch index: {e}")) - })?; - } - // vecq assigns rows sequentially; row == key by construction - // (key was derived from `index.len()` above; dims validated up front). - #[cfg(feature = "vecq")] - self.index.add(embedding); + // vecq assigns rows sequentially; row == key by construction. + self.engine.add(key, embedding)?; self.dirty = true; Ok(()) @@ -444,10 +746,7 @@ impl VectorIndex { pub fn remove(&mut self, id: &str) -> bool { if let Some(key) = self.id_to_key.remove(id) { self.key_to_id.remove(&key); - #[cfg(feature = "usearch")] - if let Err(e) = self.index.remove(key) { - tracing::error!("Failed to remove from usearch index: {e}"); - } + self.engine.remove(key); // vecq: tombstone is implicit — the key vanishes from the map, so // search results referencing that row are filtered out below. self.dirty = true; @@ -468,65 +767,38 @@ impl VectorIndex { let count = k.max(1); - #[cfg(feature = "usearch")] - let results = match self.index.search(query, count) { - Ok(r) => r, - Err(e) => { - tracing::error!("usearch search failed: {e}"); - return Vec::new(); - } - }; - #[cfg(feature = "usearch")] - return results - .keys - .iter() - .zip(results.distances.iter()) - .filter_map(|(key, dist)| self.key_to_id.get(key).map(|id| (id.clone(), *dist))) + // vecq: tombstoned rows (physically present but absent from the key + // map) are filtered out below — over-fetch by the exact dead-row + // count so we still return up to `k` live results. usearch: len() == + // live size, dead = 0. + let dead = self.engine.len().saturating_sub(self.key_to_id.len()); + let results: Vec<(String, f32)> = self + .engine + .search(query, count + dead) + .into_iter() + .filter_map(|(key, dist)| self.key_to_id.get(&key).map(|id| (id.clone(), dist))) .collect(); - - // vecq backend (#1098): brute-force top-k over quantized codes, - // returns (row, cosine similarity). Rows not present in key_to_id are - // tombstoned and filtered out. Convert similarity → cosine distance - // (1 - sim) so downstream scoring matches the usearch backend. - #[cfg(feature = "vecq")] - { - // Tombstoned rows (physically present in the index but absent - // from the key map) are filtered out below — over-fetch by the - // exact dead-row count so we still return up to `k` live results. - let dead = self.index.len().saturating_sub(self.key_to_id.len()); - let mut results: Vec<(String, f32)> = self - .index - .search(query, count + dead) - .into_iter() - .filter_map(|(row, sim)| { - self.key_to_id - .get(&(row as u64)) - .map(|id| (id.clone(), 1.0 - sim)) - }) - .collect(); - // Tombstones may shrink results below k; over-fetch above mitigates - // this. Cap at k and keep ascending-distance order. - results.truncate(count); - results - } + // Tombstones may shrink results below k; over-fetch above mitigates + // this. Cap at k and keep ascending-distance order. + let mut results = results; + results.truncate(count); + results } /// Number of live (non-tombstoned) items in the index. #[allow(dead_code)] pub fn len(&self) -> usize { #[cfg(feature = "usearch")] - return self.key_to_id.len().max(self.index.size()); - #[cfg(feature = "vecq")] - return self.key_to_id.len(); + if self.backend == VectorBackend::Usearch { + return self.key_to_id.len().max(self.engine.len()); + } + self.key_to_id.len() } /// Capacity of the underlying index (diagnostics). #[allow(dead_code)] pub fn capacity(&self) -> usize { - #[cfg(feature = "usearch")] - return self.index.capacity(); - #[cfg(feature = "vecq")] - return self.index.len(); + self.engine.capacity() } /// Embedding dimensionality of this index. @@ -534,10 +806,7 @@ impl VectorIndex { /// Used by backend dispatch to detect dim mismatch when the user swaps /// embedding backends on an existing store (#337). pub fn dims(&self) -> usize { - #[cfg(feature = "usearch")] - return self.index.dimensions(); - #[cfg(feature = "vecq")] - return self.index.dim(); + self.engine.dims() } /// Check if the index is empty. @@ -550,27 +819,6 @@ impl VectorIndex { pub fn is_dirty(&self) -> bool { self.dirty } - - #[cfg(feature = "usearch")] - fn create_index(dims: usize) -> Result { - let options = IndexOptions { - dimensions: dims, - metric: MetricKind::Cos, - quantization: ScalarKind::F32, - ..Default::default() - }; - - Index::new(&options).map_err(|e| { - Error::embed_msg(format!( - "Failed to create usearch index (dims={dims}): {e}. This is likely an out-of-memory condition." - )) - }) - } - - #[cfg(feature = "vecq")] - fn create_index(dims: usize) -> Result { - Ok(VecqIndex::new(dims, VECQ_SEED)) - } } impl Default for VectorIndex { @@ -790,7 +1038,7 @@ mod tests { // Verify the saved file is loadable by the backend's buffer API let buffer = std::fs::read(&path).unwrap(); #[cfg(feature = "usearch")] - { + if idx.backend() == VectorBackend::Usearch { let raw_index = usearch::Index::restore_from_buffer(&buffer); assert!( raw_index.is_ok(), @@ -799,7 +1047,7 @@ mod tests { assert_eq!(raw_index.unwrap().size(), 1); } #[cfg(feature = "vecq")] - { + if idx.backend() == VectorBackend::Vecq { let raw_index = VecqIndex::from_bytes(&buffer); assert!( raw_index.is_ok(), @@ -834,7 +1082,7 @@ mod tests { fn test_vecq_tombstones_filter_removed_rows() { // Removed IDs must never appear in search results (vecq has no // incremental delete — rows are tombstoned via the key map, #1098). - let mut idx = VectorIndex::new(64).unwrap(); + let mut idx = VectorIndex::with_backend(VectorBackend::Vecq, 64).unwrap(); let v1 = make_vec(64, 0); let v2 = make_vec(64, 1); @@ -846,4 +1094,72 @@ mod tests { assert!(results.iter().all(|(id, _)| id != "dead")); assert!(results.iter().any(|(id, _)| id == "alive")); } + + /// #1168: runtime selection metadata. + #[test] + fn test_backend_selection_metadata() { + // default_backend is one of the compiled-in engines + assert!(VectorBackend::default_backend().is_compiled_in()); + + // resolve: None → default, not honored + let (resolved, honored) = VectorBackend::resolve(None); + assert_eq!(resolved, VectorBackend::default_backend()); + assert!(!honored); + + // resolve: compiled-in preference honored + if VectorBackend::Vecq.is_compiled_in() { + let (resolved, honored) = VectorBackend::resolve(Some(VectorBackend::Vecq)); + assert_eq!(resolved, VectorBackend::Vecq); + assert!(honored); + } + if VectorBackend::Usearch.is_compiled_in() { + let (resolved, honored) = VectorBackend::resolve(Some(VectorBackend::Usearch)); + assert_eq!(resolved, VectorBackend::Usearch); + assert!(honored); + } + + // parse: case-insensitive, whitespace tolerant + assert_eq!(VectorBackend::parse(" vecq "), Some(VectorBackend::Vecq)); + assert_eq!( + VectorBackend::parse("USEARCH"), + Some(VectorBackend::Usearch) + ); + assert_eq!(VectorBackend::parse("bogus"), None); + } + + /// #1168: with_backend creates the requested engine (when compiled in). + #[test] + fn test_with_backend_creates_requested_engine() { + if VectorBackend::Vecq.is_compiled_in() { + let idx = VectorIndex::with_backend(VectorBackend::Vecq, 768).unwrap(); + assert_eq!(idx.backend(), VectorBackend::Vecq); + } + if VectorBackend::Usearch.is_compiled_in() { + let idx = VectorIndex::with_backend(VectorBackend::Usearch, 768).unwrap(); + assert_eq!(idx.backend(), VectorBackend::Usearch); + } + } + + /// #1168 (dual-engine builds): each engine round-trips through its own + /// per-extension file, and both files can coexist independently. + #[cfg(all(feature = "usearch", feature = "vecq"))] + #[test] + fn test_dual_engine_roundtrip_per_extension() { + let dir = tempfile::tempdir().unwrap(); + let v = make_vec(64, 3); + + for backend in [VectorBackend::Usearch, VectorBackend::Vecq] { + let path = dir.path().join(format!("idx.{}", index_ext_for(backend))); + let mut idx = VectorIndex::with_backend(backend, 64).unwrap(); + idx.path = Some(path.clone()); + idx.insert(&format!("mem-{backend:?}"), &v).unwrap(); + idx.save().unwrap(); + + let loaded = VectorIndex::load(&path).unwrap(); + assert_eq!(loaded.backend(), backend, "ext must select the engine"); + assert_eq!(loaded.len(), 1); + let results = loaded.search(&v, 1, 50); + assert_eq!(results.len(), 1); + } + } } diff --git a/crates/uteke-core/src/operations.rs b/crates/uteke-core/src/operations.rs index eec39985..f59eb5b2 100644 --- a/crates/uteke-core/src/operations.rs +++ b/crates/uteke-core/src/operations.rs @@ -2,7 +2,8 @@ use crate::error::Error; use crate::memory::types::{ - BulkDeleteResult, DEFAULT_NAMESPACE, Memory, MemoryTier, RecallStrategy, SearchResult, TagInfo, + BulkDeleteResult, DEFAULT_NAMESPACE, Memory, MemoryTier, NamespaceDeleteResult, + NamespaceRenameResult, RecallStrategy, SearchResult, TagInfo, }; use crate::memory::vector::cosine_distance_to_similarity; use std::sync::Mutex; @@ -179,17 +180,31 @@ impl crate::Uteke { } else { content.to_string() }; - // Lazy-load embedder on first use - self.ensure_embedder()?; - // Retry embedding generation up to 3 times with exponential backoff. - // Embedding failures silently drop vector entries, causing desync (#621). - let embedding = self::retry_embed(&self.embedder, &embed_text)?; + // Lazy-load embedder on first use. #1166: when no backend is + // configured (backend == "" on a build without the onnx feature, + // or open_with_backend(.., None)), skip embedding entirely — the + // row is stored FTS5-only and stays keyword-searchable. Vector + // search for this row becomes available once an embedding is + // supplied via the injected-embedding path or `uteke repair`. + let embedding = if self.embedder_backend.is_empty() { + tracing::debug!("No embedding backend configured; storing memory FTS5-only (#1166)"); + Vec::new() + } else { + self.ensure_embedder()?; + // Retry embedding generation up to 3 times with exponential backoff. + // Embedding failures silently drop vector entries, causing desync (#621). + self::retry_embed(&self.embedder, &embed_text)? + }; // Dedup check: if an existing memory has cosine >= 0.95, return it // instead of creating a duplicate (#442 enhancement). - if let Some(existing_id) = self.check_duplicate(&embedding, namespace)? { - tracing::info!("Dedup: memory {existing_id} is nearly identical, skipping insert"); - return Ok(existing_id); + // #1166: skip when no embedding was produced (no embedder) — cosine + // dedup is meaningless without vectors. + if !embedding.is_empty() { + if let Some(existing_id) = self.check_duplicate(&embedding, namespace)? { + tracing::info!("Dedup: memory {existing_id} is nearly identical, skipping insert"); + return Ok(existing_id); + } } self.remember_precomputed( @@ -315,6 +330,17 @@ impl crate::Uteke { self.store.insert(&memory)?; + // Provenance: record the content hash at write time (#1172 Fase 1). + // Best-effort — audits recompute this to detect post-write tampering. + use sha2::Digest; + let source_hash: String = sha2::Sha256::digest(content.as_bytes()) + .iter() + .map(|b| format!("{b:02x}")) + .collect(); + if let Err(e) = self.store.set_source_hash(&id, Some(source_hash.as_str())) { + tracing::warn!("source_hash write failed for {id}: {e}"); + } + // Timeline: record creation (#347). This hook lives in the single // shared creation path so every remember() / remember_typed() / // remember_precomputed() / consolidate() call records a Created @@ -334,7 +360,13 @@ impl crate::Uteke { // Invalidate recall cache — new memory may affect future queries self.recall_cache.invalidate_namespace(&memory.namespace); - index.insert(&id, embedding)?; + // #1166: empty embedding = "no embedder configured" — store the row + // (already committed to SQLite above) without a vector entry. The + // row stays FTS5-searchable; `uteke repair` can backfill vectors + // once an embedder is available. + if !embedding.is_empty() { + index.insert(&id, embedding)?; + } // Retry index persistence up to 3 times (#621). // A failed save means the in-memory index has the entry but // on-disk doesn't → silent desync on next process launch. @@ -365,7 +397,11 @@ impl crate::Uteke { // Cosine-similarity auto-linking (#401). // Must run AFTER index.insert() so the new memory is searchable. // Best-effort: errors logged, never fails remember(). - self.auto_link_cosine(&id, embedding, Some(memory.namespace.as_str())); + // #1166: cosine auto-linking needs a real embedding; skip when the + // row was stored FTS5-only (no embedder configured). + if !embedding.is_empty() { + self.auto_link_cosine(&id, embedding, Some(memory.namespace.as_str())); + } Ok(id) } @@ -640,7 +676,9 @@ impl crate::Uteke { /// /// Callers must NOT cache inside this method — the dispatcher owns the /// cache put and applies salience/recency boosts exactly once. - fn compute_recall( + /// `pub(crate)` for the #1160 explanation path, which replays the same + /// building blocks stage-by-stage. + pub(crate) fn compute_recall( &self, strategy: RecallStrategy, query: &str, @@ -1352,6 +1390,187 @@ impl crate::Uteke { self.store.list_namespaces_with_counts() } + /// List all namespaces with counts split by lifecycle state (#1181). + /// + /// Returns `[(namespace, active, deprecated)]` so clients can show honest + /// breakdowns — deprecated-only "ghost" namespaces look identical to + /// active ones in plain counts. + pub fn list_namespaces_with_lifecycle_counts( + &self, + ) -> Result, Error> { + self.store.list_namespaces_with_lifecycle_counts() + } + + /// Move a single memory to another namespace (#1181). + /// + /// Returns `Ok(false)` when the memory does not exist. Namespace is a + /// plain column and embeddings are content-based, so no re-embed happens. + pub fn move_memory(&self, id: &str, namespace: &str) -> Result { + Self::validate_namespace_name(namespace)?; + let existing = match self.store.get_by_id(id)? { + Some(memory) => memory, + None => return Ok(false), + }; + let moved = + self.store + .move_memory_namespace(id, namespace, &chrono::Utc::now().to_rfc3339())?; + if moved { + self.recall_cache.invalidate_namespace(&existing.namespace); + self.recall_cache.invalidate_namespace(namespace); + } + Ok(moved) + } + + /// Rename a namespace, merging into the target when it exists (#1181). + /// + /// Single atomic `UPDATE` — the old name vanishes naturally because + /// namespaces are a derived view over `memories.namespace`. + pub fn rename_namespace(&self, from: &str, to: &str) -> Result { + Self::validate_namespace_name(from)?; + Self::validate_namespace_name(to)?; + if from == to { + return Err(Error::Validation( + "Rename source and target namespaces are identical".to_string(), + )); + } + let (active, deprecated, total) = self.store.namespace_lifecycle_counts(from)?; + if total == 0 { + return Err(Error::Validation(format!("Namespace not found: {from}"))); + } + let target_existed = self.store.namespace_lifecycle_counts(to)?.2 > 0; + let moved = self + .store + .rename_namespace(from, to, &chrono::Utc::now().to_rfc3339())?; + self.recall_cache.invalidate_namespace(from); + self.recall_cache.invalidate_namespace(to); + tracing::info!( + "Namespace rename/merge: '{from}' -> '{to}' moved {moved} memories \ + ({active} active, {deprecated} deprecated, target_existed={target_existed})" + ); + Ok(NamespaceRenameResult { + from: from.to_string(), + to: to.to_string(), + moved, + target_existed, + }) + } + + /// Delete a namespace with an explicit strategy for its memories (#1181). + /// + /// Namespaces are derived — the name only disappears once no memory + /// references it, so deletion must first decide the fate of its memories: + /// - `refuse`: fail while any memory (incl. deprecated) uses the name. + /// - `merge`: move all memories into `target` (the name disappears). + /// - `deprecate`: soft-delete all memories (recycle bin + TTL) — the name + /// stays visible as a deprecated-only ghost with honest counts. + /// + /// There is no hard-delete path, consistent with the lifecycle design. + pub fn delete_namespace( + &self, + name: &str, + strategy: &str, + target: Option<&str>, + ) -> Result { + Self::validate_namespace_name(name)?; + match strategy { + "refuse" | "merge" | "deprecate" => {} + other => { + return Err(Error::Validation(format!( + "Unknown delete strategy '{other}'. Expected: refuse, merge, deprecate" + ))); + } + } + let (active, deprecated, total) = self.store.namespace_lifecycle_counts(name)?; + let no_op = |empty: bool| NamespaceDeleteResult { + name: name.to_string(), + strategy: strategy.to_string(), + affected: 0, + target: target.map(str::to_string), + empty, + }; + if total == 0 { + // Namespace is empty / already gone — idempotent no-op. + return Ok(no_op(true)); + } + match strategy { + "merge" => { + let target = target.ok_or_else(|| { + Error::Validation("strategy=merge requires a 'target' namespace".to_string()) + })?; + Self::validate_namespace_name(target)?; + if target == name { + return Err(Error::Validation( + "Merge target must differ from the deleted namespace".to_string(), + )); + } + let moved = + self.store + .rename_namespace(name, target, &chrono::Utc::now().to_rfc3339())?; + self.recall_cache.invalidate_namespace(name); + self.recall_cache.invalidate_namespace(target); + tracing::info!( + "Namespace delete (merge): '{name}' -> '{target}' moved {moved} memories" + ); + Ok(NamespaceDeleteResult { + name: name.to_string(), + strategy: strategy.to_string(), + affected: moved, + target: Some(target.to_string()), + empty: true, + }) + } + "deprecate" => { + let reason = format!("namespace delete (strategy=deprecate) of '{name}' (#1181)"); + let affected = self.store.deprecate_by_namespace(name, &reason)?; + let ids = self.store.namespace_ids(name)?; + let mut index = self + .index + .write() + .map_err(|_| Error::lock("index write lock during delete_namespace"))?; + for id in &ids { + index.remove(id); + } + persist_index_after_delete(&mut index, "delete_namespace (deprecate)")?; + self.recall_cache.invalidate_namespace(name); + tracing::info!( + "Namespace delete (deprecate): '{name}' soft-deleted {affected} memories \ + ({deprecated} were already deprecated)" + ); + Ok(NamespaceDeleteResult { + name: name.to_string(), + strategy: strategy.to_string(), + affected, + target: None, + // The name survives as a deprecated-only ghost listing. + empty: false, + }) + } + // `refuse` — validated above; total > 0 always refuses. + _ => Err(Error::Validation(format!( + "Namespace '{name}' still holds {total} memory(ies) \ + ({active} active, {deprecated} deprecated); refusing to delete. \ + Use strategy=merge (with target) or strategy=deprecate." + ))), + } + } + + /// Shared namespace name validation (#1181). + fn validate_namespace_name(name: &str) -> Result<(), Error> { + let trimmed = name.trim(); + if trimmed.is_empty() { + return Err(Error::Validation( + "Namespace name must not be empty".to_string(), + )); + } + if trimmed.len() > 128 { + return Err(Error::Validation(format!( + "Namespace name too long ({} > 128 chars)", + trimmed.len() + ))); + } + Ok(()) + } + /// List all tags with their usage counts. pub fn tags_with_counts(&self, namespace: Option<&str>) -> Result, Error> { self.store.tags_with_counts(namespace) @@ -2016,16 +2235,17 @@ mod recall_cache_parity_tests { /// Fusion weights benchmark-tuned on LongMemEval fast50 (#1123): /// plateau [1.7, 1.9] → R@5 0.98; 1.7 chosen mid-plateau. /// k=60 matches the k used by recall_rrf. -const FUSION_W_VECTOR: f64 = 1.7; -const FUSION_W_HYBRID: f64 = 1.0; -const FUSION_RRF_K: f64 = 60.0; +pub(crate) const FUSION_W_VECTOR: f64 = 1.7; +pub(crate) const FUSION_W_HYBRID: f64 = 1.0; +pub(crate) const FUSION_RRF_K: f64 = 60.0; /// Weighted RRF fuse of two complete SearchResult rankings (#1123). /// /// Deduplicates by memory id (first occurrence keeps the Memory payload), /// sorts by fused score descending, and rewrites each result's score to the /// fused RRF score. No truncation — the caller truncates to its window. -fn rrf_fuse_weighted( +/// `pub(crate)` for the #1160 explanation path, which replays the same fuse. +pub(crate) fn rrf_fuse_weighted( primary: Vec, secondary: Vec, w_primary: f64, @@ -2402,3 +2622,207 @@ pub fn memory_existed_at( } true } + +#[cfg(test)] +mod namespace_management_tests { + use crate::Uteke; + + /// #1181: move_memory updates the namespace column and both the old and + /// the new namespace disappear/appear correctly in listings. + #[test] + fn move_memory_updates_namespace() { + let u = Uteke::open_with_backend(":memory:", None).expect("open without embedder"); + let id = u + .remember("move me", &[], None, Some("alpha")) + .expect("remember"); + + let moved = u.move_memory(&id, "beta").expect("move"); + assert!(moved); + let mem = u.get_by_id(&id).expect("get").expect("exists"); + assert_eq!(mem.namespace, "beta"); + + let listings = u.list_namespaces().expect("list"); + assert!( + !listings.contains(&"alpha".to_string()), + "old name vanishes" + ); + assert!(listings.contains(&"beta".to_string())); + + // Unknown ID → Ok(false), not an error. + let missing = u.move_memory("00000000-0000-4000-8000-000000000000", "beta"); + assert!(matches!(missing, Ok(false))); + } + + /// #1181: rename moves all memories; renaming onto an existing namespace + /// merges and reports `target_existed = true`. + #[test] + fn rename_namespace_moves_and_merges() { + let u = Uteke::open_with_backend(":memory:", None).expect("open without embedder"); + u.remember("a one", &[], None, Some("old")).expect("a1"); + u.remember("a two", &[], None, Some("old")).expect("a2"); + u.remember("b one", &[], None, Some("new")).expect("b1"); + + // Merge path: target exists. + let merge = u.rename_namespace("old", "new").expect("merge"); + assert!(merge.target_existed); + assert_eq!(merge.moved, 2); + assert_eq!(merge.from, "old"); + assert_eq!(merge.to, "new"); + + let listings = u.list_namespaces().expect("list"); + assert!(!listings.contains(&"old".to_string())); + assert!(listings.contains(&"new".to_string())); + + // Plain rename path: target does not exist. + u.remember("c one", &[], None, Some("temp")).expect("c1"); + let plain = u.rename_namespace("temp", "final").expect("rename"); + assert!(!plain.target_existed); + assert_eq!(plain.moved, 1); + + // Same-name rename is rejected. + assert!(u.rename_namespace("final", "final").is_err()); + // Unknown source namespace is rejected. + assert!(u.rename_namespace("ghost", "anywhere").is_err()); + } + + /// #1181: delete strategies — refuse (default) blocks while memories + /// remain; merge moves everything away; deprecate soft-deletes without + /// hard-deleting anything. + #[test] + fn delete_namespace_strategies() { + let u = Uteke::open_with_backend(":memory:", None).expect("open without embedder"); + let id1 = u.remember("d one", &[], None, Some("temp-ns")).expect("d1"); + let id2 = u.remember("d two", &[], None, Some("temp-ns")).expect("d2"); + + // refuse (default): blocked while memories exist. + let refused = u.delete_namespace("temp-ns", "refuse", None); + assert!(refused.is_err(), "refuse must block a non-empty namespace"); + + // merge: moves all memories into the target, name disappears. + let merged = u + .delete_namespace("temp-ns", "merge", Some("archive")) + .expect("merge delete"); + assert_eq!(merged.strategy, "merge"); + assert_eq!(merged.affected, 2); + assert_eq!(merged.target.as_deref(), Some("archive")); + assert!(merged.empty); + let listings = u.list_namespaces().expect("list"); + assert!(!listings.contains(&"temp-ns".to_string())); + assert!(u.get_by_id(&id1).expect("get").expect("survives").namespace == "archive"); + + // deprecate: soft-delete only, no hard delete; the name remains as a + // deprecated-only ghost with honest lifecycle counts. + let dep1 = u + .remember("e one", &[], None, Some("ghost-ns")) + .expect("e1"); + let dep = u + .delete_namespace("ghost-ns", "deprecate", None) + .expect("deprecate delete"); + assert_eq!(dep.strategy, "deprecate"); + assert_eq!(dep.affected, 1); + assert!(!dep.empty); + let mem = u.get_by_id(&dep1).expect("get").expect("still stored"); + assert!(mem.deprecated, "memory must be soft-deleted, not removed"); + let lifecycle = u + .list_namespaces_with_lifecycle_counts() + .expect("lifecycle counts"); + let ghost = lifecycle + .iter() + .find(|(name, _, _)| name == "ghost-ns") + .expect("ghost namespace stays listed"); + assert_eq!((ghost.1, ghost.2), (0, 1), "0 active / 1 deprecated"); + assert!( + u.get_by_id(&id2).is_ok(), + "nothing from the earlier merge path was lost" + ); + + // Unknown strategy is rejected. + assert!(u.delete_namespace("ghost-ns", "hard-delete", None).is_err()); + } + + /// #1172 Fase 1: remember records a SHA-256 source hash; provenance() + /// returns the full chain (fields + tier + timeline) and detects content + /// modified after write via hash mismatch. + #[test] + fn provenance_chain_and_source_hash() { + use sha2::Digest; + + let u = Uteke::open_with_backend(":memory:", None).expect("open without embedder"); + let id = u + .remember("the deploy window is 09:00 WIB", &[], None, Some("ops")) + .expect("remember"); + + let report = u.provenance(&id).expect("provenance").expect("exists"); + assert_eq!(report.id, id); + assert_eq!(report.namespace, "ops"); + assert_eq!(report.author_type, "agent"); + // Hash at write time must match a live recomputation. + let expected: String = sha2::Sha256::digest(report.content.as_bytes()) + .iter() + .map(|b| format!("{b:02x}")) + .collect(); + assert_eq!(report.source_hash.as_deref(), Some(expected.as_str())); + assert_eq!(report.content_hash_now, expected); + assert!(!report.events.is_empty(), "Created event must be recorded"); + assert!(report.events.iter().any(|e| e.event_type == "created")); + + // Update content without touching source_hash → mismatch detected + // (this is the tamper-evidence property). + u.store + .conn + .execute( + "UPDATE memories SET content = 'tampered' WHERE id = ?1", + rusqlite::params![id], + ) + .expect("tamper"); + let after = u.provenance(&id).expect("provenance").expect("exists"); + assert_eq!(after.content, "tampered"); + assert_ne!( + after.source_hash.as_deref(), + Some(after.content_hash_now.as_str()), + "post-write modification must break the hash match" + ); + + // Unknown ID → Ok(None). + assert!( + u.provenance("00000000-0000-4000-8000-000000000000") + .unwrap() + .is_none() + ); + } + + /// #1172 Fase 1: timeline events carry actor + evidence when written via + /// the provenance-aware API, and old callers (no provenance) stay working. + #[test] + fn timeline_events_carry_provenance() { + let u = Uteke::open_with_backend(":memory:", None).expect("open without embedder"); + let id = u + .remember("evidence chain probe", &[], None, None) + .expect("remember"); + + u.store + .add_timeline_event_with_provenance( + &id, + crate::timeline::TimelineEventType::Updated, + Some(&serde_json::json!({"field": "importance"})), + Some("agent:cto"), + Some(&serde_json::json!([{"memory": "other-id", "score": 0.82}])), + ) + .expect("append provenance event"); + + let events = u.timeline(&id, 0).expect("timeline"); + assert_eq!(events.len(), 2, "created + updated"); + let provenance_event = events + .iter() + .find(|e| e.event_type == "updated") + .expect("updated event"); + assert_eq!(provenance_event.actor.as_deref(), Some("agent:cto")); + let evidence = provenance_event.evidence.as_ref().expect("evidence"); + assert_eq!(evidence[0]["memory"], serde_json::json!("other-id")); + + // Plain events (Created) have no actor — backward compatible shape. + let created = events.iter().find(|e| e.event_type == "created").unwrap(); + assert!(created.actor.is_none()); + assert!(created.evidence.is_none()); + } +} diff --git a/crates/uteke-core/src/provenance.rs b/crates/uteke-core/src/provenance.rs index 0fcb1335..5247a18e 100644 --- a/crates/uteke-core/src/provenance.rs +++ b/crates/uteke-core/src/provenance.rs @@ -131,6 +131,70 @@ fn contains_hedge(lower: &str) -> bool { HEDGE_MARKERS.iter().any(|h| lower.contains(h)) } +// ── Provenance chain query (#1172 Fase 1) ────────────────────────────────── + +/// Full provenance view of a memory (#1172 Fase 1): identity + provenance +/// fields + trust tier + content hash + the auditable event chain. +#[derive(Debug, Clone, Serialize, Deserialize)] +pub struct ProvenanceReport { + pub id: String, + pub namespace: String, + pub created_at: String, + pub updated_at: String, + pub author_type: String, + pub source: Option, + pub source_type: String, + /// SHA-256 of the content at the last write (schema v18+). `None` for + /// memories written before the column existed. + pub source_hash: Option, + /// Content currently stored — lets an auditor recompute the hash and + /// verify tamper-evidence. + pub content: String, + /// Hash of the current content (computed live) — compare against + /// `source_hash` to detect post-write modifications. + pub content_hash_now: String, + pub trust_tier: TrustTier, + pub deprecated: bool, + /// Event chain, newest first, with actor + evidence when recorded. + pub events: Vec, +} + +impl crate::Uteke { + /// Build the full provenance report for a memory (#1172 Fase 1). + /// + /// Returns `Ok(None)` when the memory does not exist. + pub fn provenance(&self, id: &str) -> Result, crate::Error> { + use sha2::Digest; + + let memory = match self.store.get_by_id(id)? { + Some(m) => m, + None => return Ok(None), + }; + let source_hash = self.store.get_source_hash(id)?; + let content_hash_now: String = sha2::Sha256::digest(memory.content.as_bytes()) + .iter() + .map(|b| format!("{b:02x}")) + .collect(); + let events = self.store.list_timeline_events(id, 0)?; + let trust_tier = TrustTier::of(&memory); + Ok(Some(ProvenanceReport { + id: memory.id.clone(), + namespace: memory.namespace.clone(), + created_at: memory.created_at.to_rfc3339(), + updated_at: memory.updated_at.to_rfc3339(), + author_type: memory.author_type.clone(), + source: memory.source.clone(), + source_type: memory.source_type.clone(), + source_hash, + content: memory.content, + content_hash_now, + trust_tier, + deprecated: memory.deprecated, + events, + })) + } +} + #[cfg(test)] mod tests { use super::*; diff --git a/crates/uteke-core/src/recall_explain.rs b/crates/uteke-core/src/recall_explain.rs new file mode 100644 index 00000000..0a171690 --- /dev/null +++ b/crates/uteke-core/src/recall_explain.rs @@ -0,0 +1,733 @@ +//! Recall explanation / debug mode (#1160). +//! +//! `Uteke::recall_explained` runs the SAME building blocks as the normal +//! recall path (vector channel, FTS5 channel, weighted-RRF, Jaccard boost, +//! salience/recency boosts, graph rerank) but instruments every stage, so +//! each returned result carries the signals that placed it. +//! +//! Deliberate trade-offs (documented in the issue): +//! - Cold compute only — the recall cache is bypassed so the explanation +//! always describes the results actually returned. +//! - One extra query embedding (~50ms) + one extra FTS5 query vs a normal +//! cold call; no additional index searches or model loads. + +use std::collections::HashMap; + +use serde::Serialize; + +use crate::Error; +use crate::Uteke; +use crate::memory::types::{Memory, RecallStrategy, SearchResult}; +use crate::operations::{FUSION_RRF_K, FUSION_W_HYBRID, FUSION_W_VECTOR}; + +/// Per-result ranking signal breakdown (#1160). +#[derive(Debug, Clone, Serialize)] +pub struct RecallExplanation { + /// Strategy that produced this result. + pub strategy: String, + /// The final score the result ranked on (post-boosts). + pub final_score: f32, + /// Score before salience/recency boosts (after RRF/jaccard/graph). + pub base_score: f32, + /// Raw cosine similarity between the query embedding and the memory + /// embedding. `None` when the memory has no embedding. + pub vector_similarity: Option, + /// 1-based rank in the vector channel ranking. `None` when the memory + /// was outside the vector channel's candidate window. + pub vector_rank: Option, + /// 1-based rank in the FTS5 channel ranking. `None` when absent. + pub fts_rank: Option, + /// Normalized RRF score (before jaccard/boosts) — hybrid and fusion. + pub rrf_score: Option, + /// Fusion only: weighted RRF contribution of the vector channel. + pub fusion_vector_contribution: Option, + /// Fusion only: weighted RRF contribution of the hybrid channel. + pub fusion_hybrid_contribution: Option, + /// Jaccard token-overlap boost added to the base score (hybrid/graph). + pub jaccard_boost: Option, + /// Salience boost delta included in the final score. + pub salience_boost: Option, + /// Recency boost delta included in the final score. + pub recency_boost: Option, + /// Graph-strategy only: total graph rerank delta. + pub graph_boost: Option, +} + +/// A recall result with its full ranking explanation (#1160). +/// +/// Serialized with `result` flattened so JSON consumers see the familiar +/// `{memory, score}` shape plus an additive `explanation` object. +#[derive(Debug, Clone, Serialize)] +pub struct ExplainedRecall { + #[serde(flatten)] + pub result: SearchResult, + pub explanation: RecallExplanation, +} + +/// RRF rank→score contribution: `weight / (k + rank)` with a 1-based rank. +fn rrf_contrib(weight: f64, rank: usize) -> f64 { + weight / (FUSION_RRF_K + rank as f64) +} + +/// Raw cosine similarity between two vectors (0.0 for zero-norm inputs). +fn cosine_similarity(a: &[f32], b: &[f32]) -> f32 { + let dot: f32 = a.iter().zip(b.iter()).map(|(x, y)| x * y).sum(); + let na: f32 = a.iter().map(|x| x * x).sum::().sqrt(); + let nb: f32 = b.iter().map(|x| x * x).sum::().sqrt(); + if na <= 0.0 || nb <= 0.0 { + 0.0 + } else { + (dot / (na * nb)).clamp(0.0, 1.0) + } +} + +/// Rank of each memory id in a channel ranking (1-based, first occurrence). +fn rank_map<'a, I>(items: I) -> HashMap +where + I: Iterator, +{ + items + .enumerate() + .map(|(i, sr)| (sr.memory.id.clone(), i + 1)) + .collect() +} + +impl Uteke { + /// Recall with a per-result ranking explanation (#1160). + /// + /// Mirrors `recall_hybrid` stage-for-stage (same channel depths, same + /// RRF constants, same boost order) but bypasses the recall cache so the + /// explanation always matches the returned results. + pub fn recall_explained( + &self, + query: &str, + limit: usize, + tags_filter: Option<&[&str]>, + namespace: Option<&str>, + strategy: RecallStrategy, + min_score: f32, + ) -> Result, Error> { + // Fts5 needs no query embedding — keep it usable without an embedder + // (CI-safe, same contract as the fts5 strategy itself). All other + // strategies embed the query once, exactly like the real path. + let query_embedding: Option> = if strategy == RecallStrategy::Fts5 { + None + } else { + self.ensure_embedder()?; + Some( + self.embedder + .lock() + .map_err(|_| Error::lock("embedder lock during recall_explained"))? + .as_ref() + .ok_or_else(|| Error::embed_msg("no embedder configured"))? + .embed(query)?, + ) + }; + + let boost_window = limit.saturating_mul(4).saturating_add(16); + + let strategy_name = match strategy { + RecallStrategy::Vector => "vector", + RecallStrategy::Fts5 => "fts5", + RecallStrategy::Hybrid => "hybrid", + RecallStrategy::Graph => "graph", + RecallStrategy::Fusion => "fusion", + }; + + // Cosine similarity helper over the query embedding (None when the + // strategy skips embedding or the memory has no vector). + let vec_sim_of = |m: &Memory| -> Option { + let qe = query_embedding.as_ref()?; + if m.embedding.is_empty() { + return None; + } + Some(cosine_similarity(qe, &m.embedding)) + }; + + /// Intermediate signals captured while the strategy pipeline is + /// reconstructed from the same building blocks. + struct Sig { + base: f32, + vector_similarity: Option, + vector_rank: Option, + fts_rank: Option, + rrf_score: Option, + fusion_vector_contribution: Option, + fusion_hybrid_contribution: Option, + jaccard_boost: Option, + graph_boost: Option, + } + let empty_sig = |base: f32| Sig { + base, + vector_similarity: None, + vector_rank: None, + fts_rank: None, + rrf_score: None, + fusion_vector_contribution: None, + fusion_hybrid_contribution: None, + jaccard_boost: None, + graph_boost: None, + }; + + // (ordered results with raw base scores, signals per id) + let (mut results, mut signals): (Vec, HashMap) = match strategy { + RecallStrategy::Vector => { + // The real vector arm via compute_recall (same entry point + // the dispatcher uses — keeps the cache/boost contract of + // this function consistent across strategies). + let results = self.compute_recall( + RecallStrategy::Vector, + query, + limit, + tags_filter, + namespace, + 0.0, + )?; + let ranks = rank_map(results.iter()); + let mut sigs: HashMap = HashMap::new(); + for sr in &results { + let mut s = empty_sig(sr.score); + s.vector_similarity = vec_sim_of(&sr.memory); + s.vector_rank = ranks.get(&sr.memory.id).copied(); + sigs.insert(sr.memory.id.clone(), s); + } + (results, sigs) + } + RecallStrategy::Fts5 => { + // The real fts5 arm (compute_recall::Fts5 → recall_fts5_only). + let results = self.compute_recall( + RecallStrategy::Fts5, + query, + limit, + tags_filter, + namespace, + 0.0, + )?; + let mut sigs: HashMap = HashMap::new(); + for (i, sr) in results.iter().enumerate() { + let mut s = empty_sig(sr.score); + s.vector_similarity = vec_sim_of(&sr.memory); + s.fts_rank = Some(i + 1); + sigs.insert(sr.memory.id.clone(), s); + } + (results, sigs) + } + RecallStrategy::Hybrid | RecallStrategy::Graph => { + // recall_rrf(window): sub-channels run at window*3 depth. + let depth = boost_window.saturating_mul(3); + let vec_list = self.compute_recall( + RecallStrategy::Vector, + query, + depth, + tags_filter, + namespace, + 0.0, + )?; + let fts_list: Vec<(Memory, f64)> = { + let fts = match self.store.search_fts5(query, namespace, depth) { + Ok(r) if !r.is_empty() => r, + Ok(_) => self.store.search_fts5_tokens(query, namespace, depth)?, + Err(e) => return Err(e), + }; + fts.into_iter() + .filter(|(memory, _)| { + if let Some(ns) = namespace { + if memory.namespace != ns { + return false; + } + } + if let Some(filter_tags) = tags_filter { + if !filter_tags + .iter() + .any(|ft| memory.tags.iter().any(|t| t == ft)) + { + return false; + } + } + true + }) + .collect() + }; + + let vec_rank: HashMap = vec_list + .iter() + .enumerate() + .map(|(i, sr)| (sr.memory.id.clone(), i + 1)) + .collect(); + let fts_rank: HashMap = fts_list + .iter() + .enumerate() + .map(|(i, (m, _))| (m.id.clone(), i + 1)) + .collect(); + + // RRF merge — identical math to recall_rrf (k = 60). + let max_rrf = 2.0 / (FUSION_RRF_K + 1.0); + let mut rrf_scores: HashMap = HashMap::new(); + for (id, r) in &vec_rank { + *rrf_scores.entry(id.clone()).or_default() += 1.0 / (FUSION_RRF_K + *r as f64); + } + for (id, r) in &fts_rank { + *rrf_scores.entry(id.clone()).or_default() += 1.0 / (FUSION_RRF_K + *r as f64); + } + let mut scored: Vec<(String, f64)> = rrf_scores.into_iter().collect(); + scored.sort_by(|a, b| b.1.partial_cmp(&a.1).unwrap_or(std::cmp::Ordering::Equal)); + + // Payloads from the channel lists (first occurrence wins, + // vector channel first — same order as recall_rrf). + let mut payloads: HashMap = HashMap::new(); + for sr in &vec_list { + payloads + .entry(sr.memory.id.clone()) + .or_insert_with(|| sr.memory.clone()); + } + for (m, _) in &fts_list { + payloads.entry(m.id.clone()).or_insert_with(|| m.clone()); + } + + // take(boost_window) — matches recall_rrf's truncation. + let ordered: Vec<(String, f64)> = scored.into_iter().take(boost_window).collect(); + // Jaccard boost — identical math to recall_rrf (#719). + let jaccard_of = |m: &Memory| -> Option { + if self.jaccard_weight <= 0.0 { + return None; + } + let qt = crate::jaccard::tokenize(query); + if qt.is_empty() { + return None; + } + let mut ct = crate::jaccard::tokenize(&m.content); + for tag in &m.tags { + ct.insert(tag.to_ascii_lowercase()); + } + Some(crate::jaccard::jaccard_similarity(&qt, &ct) * self.jaccard_weight) + }; + + let mut results: Vec = Vec::with_capacity(ordered.len()); + let mut sigs: HashMap = HashMap::new(); + for (id, raw_rrf) in &ordered { + let memory = match payloads.get(id) { + Some(m) => m.clone(), + None => continue, + }; + let normalized = ((*raw_rrf / max_rrf).clamp(0.0, 1.0)) as f32; + let jb = jaccard_of(&memory); + let base = normalized + jb.unwrap_or(0.0); + let mut s = empty_sig(base); + s.vector_similarity = vec_sim_of(&memory); + s.vector_rank = vec_rank.get(id).copied(); + s.fts_rank = fts_rank.get(id).copied(); + s.rrf_score = Some(normalized); + s.jaccard_boost = jb; + sigs.insert(id.clone(), s); + results.push(SearchResult { + memory, + score: base, + }); + } + results.sort_by(|a, b| { + b.score + .partial_cmp(&a.score) + .unwrap_or(std::cmp::Ordering::Equal) + }); + + // Graph arm: hybrid RRF + graph-signal rerank delta (#378). + if strategy == RecallStrategy::Graph + && self.graph_rerank_config.enabled + && !results.is_empty() + { + let ids: Vec = results.iter().map(|r| r.memory.id.clone()).collect(); + let g_signals = + crate::graph_rerank::compute_graph_signals(&self.store.conn, &ids)?; + let before: HashMap = results + .iter() + .map(|r| (r.memory.id.clone(), r.score)) + .collect(); + results = crate::graph_rerank::rerank_with_graph( + results, + &g_signals, + &self.graph_rerank_config, + ); + for r in &results { + if let Some(s) = sigs.get_mut(&r.memory.id) { + let prev = before.get(&r.memory.id).copied().unwrap_or(r.score); + s.graph_boost = Some(r.score - prev); + s.base = r.score; + } + } + } + (results, sigs) + } + RecallStrategy::Fusion => { + // Fusion (#1123): two sub-rankings at window depth, then + // weighted RRF — the exact calls compute_recall makes. + let vec_res = self.compute_recall( + RecallStrategy::Vector, + query, + boost_window, + tags_filter, + namespace, + 0.0, + )?; + let hyb_res = self.compute_recall( + RecallStrategy::Hybrid, + query, + boost_window, + tags_filter, + namespace, + 0.0, + )?; + + let vec_rank = rank_map(vec_res.iter()); + let hyb_rank = rank_map(hyb_res.iter()); + + let fused = crate::operations::rrf_fuse_weighted( + vec_res.clone(), + hyb_res.clone(), + FUSION_W_VECTOR, + FUSION_W_HYBRID, + ); + + let mut sigs: HashMap = HashMap::new(); + for sr in &fused { + let id = &sr.memory.id; + let vr = vec_rank.get(id).copied(); + let hr = hyb_rank.get(id).copied(); + let mut s = empty_sig(sr.score); + s.vector_similarity = vec_sim_of(&sr.memory); + s.vector_rank = vr; + s.fts_rank = None; // hybrid channel exposes no FTS ranks + s.rrf_score = Some(sr.score); + s.fusion_vector_contribution = + vr.map(|r| rrf_contrib(FUSION_W_VECTOR, r) as f32); + s.fusion_hybrid_contribution = + hr.map(|r| rrf_contrib(FUSION_W_HYBRID, r) as f32); + sigs.insert(id.clone(), s); + } + (fused, sigs) + } + }; + + // Salience/recency boosts — identical math to + // apply_salience_recency_boosts, with per-axis delta capture. + let cfg = self.salience_recency_config; + if !cfg.is_noop() { + let now = chrono::Utc::now(); + for sr in results.iter_mut() { + sr.score = crate::salience_recency::apply_boosts(sr.score, &sr.memory, now, cfg); + } + } + results.sort_by(|a, b| { + b.score + .partial_cmp(&a.score) + .unwrap_or(std::cmp::Ordering::Equal) + }); + results.truncate(limit); + if min_score > 0.0 { + results.retain(|r| r.score >= min_score); + } + + // Assemble explained results; boost deltas computed per axis. + let now = chrono::Utc::now(); + let mut out = Vec::with_capacity(results.len()); + for sr in results { + let sig = signals.remove(&sr.memory.id); + let (sal, rec) = if cfg.is_noop() { + (None, None) + } else { + ( + Some(crate::salience_recency::salience_score(&sr.memory) * cfg.salience_weight), + Some( + crate::salience_recency::recency_score(&sr.memory, now) + * cfg.recency_weight, + ), + ) + }; + let explanation = match sig { + Some(s) => RecallExplanation { + strategy: strategy_name.to_string(), + final_score: sr.score, + base_score: s.base, + vector_similarity: s.vector_similarity, + vector_rank: s.vector_rank, + fts_rank: s.fts_rank, + rrf_score: s.rrf_score, + fusion_vector_contribution: s.fusion_vector_contribution, + fusion_hybrid_contribution: s.fusion_hybrid_contribution, + jaccard_boost: s.jaccard_boost, + salience_boost: sal, + recency_boost: rec, + graph_boost: s.graph_boost, + }, + None => RecallExplanation { + strategy: strategy_name.to_string(), + final_score: sr.score, + base_score: sr.score, + vector_similarity: vec_sim_of(&sr.memory), + vector_rank: None, + fts_rank: None, + rrf_score: None, + fusion_vector_contribution: None, + fusion_hybrid_contribution: None, + jaccard_boost: None, + salience_boost: sal, + recency_boost: rec, + graph_boost: None, + }, + }; + out.push(ExplainedRecall { + result: sr, + explanation, + }); + } + Ok(out) + } +} + +#[cfg(test)] +mod explain_tests { + use crate::Uteke; + use crate::memory::types::RecallStrategy; + + fn scratch() -> (Uteke, tempfile::TempDir) { + let dir = tempfile::tempdir().unwrap(); + let uteke = Uteke::open(dir.path().join("t.db")).unwrap(); + (uteke, dir) + } + + fn seed_fts(uteke: &Uteke) { + let now = chrono::Utc::now(); + let mk = |id: &str, content: &str| crate::memory::types::Memory { + id: id.to_string(), + content: content.to_string(), + embedding: vec![0.0; 768], + tags: vec![], + metadata: serde_json::json!({}), + created_at: now, + updated_at: now, + namespace: "explain-ns".to_string(), + access_count: 0, + last_accessed: None, + deprecated: false, + deprecated_at: None, + valid_from: None, + valid_until: None, + memory_type: "fact".to_string(), + importance: 0.5, + pinned: false, + content_type: "text".to_string(), + slug: None, + source: None, + source_type: "user".to_string(), + author_type: "agent".to_string(), + }; + uteke + .store + .insert(&mk( + "explain-fts-target", + "The quick brown fox jumps over the lazy dog", + )) + .unwrap(); + uteke + .store + .insert(&mk( + "explain-fts-noise", + "Completely unrelated content about gardening tools", + )) + .unwrap(); + } + + /// #1160: explained recall matches the real fts5 arm — same memory, and + /// the explanation carries the fts rank signal. Runs without an embedder. + #[test] + fn explain_fts5_matches_real_arm() { + let (uteke, dir) = scratch(); + seed_fts(&uteke); + + let plain = uteke + .recall_hybrid( + "quick brown fox", + 5, + None, + Some("explain-ns"), + RecallStrategy::Fts5, + 0.0, + ) + .unwrap(); + let explained = uteke + .recall_explained( + "quick brown fox", + 5, + None, + Some("explain-ns"), + RecallStrategy::Fts5, + 0.0, + ) + .unwrap(); + + assert!(!plain.is_empty(), "plain fts5 must find the fox"); + assert_eq!(plain[0].memory.id, "explain-fts-target"); + assert_eq!(explained.len(), plain.len(), "same result count"); + assert_eq!(explained[0].result.memory.id, plain[0].memory.id); + assert_eq!( + explained[0].result.memory.id, "explain-fts-target", + "explained fts5 must find the fox too" + ); + + let e = &explained[0].explanation; + assert_eq!(e.strategy, "fts5"); + assert_eq!( + e.fts_rank, + Some(1), + "target must be rank 1 in the fts channel" + ); + assert!( + (e.final_score - plain[0].score).abs() < 1e-6, + "explained score must equal the plain score: {} vs {}", + e.final_score, + plain[0].score + ); + + // Noise must not appear above the target. + assert!( + !explained + .iter() + .take(1) + .any(|r| r.result.memory.id == "explain-fts-noise") + ); + drop(uteke); + drop(dir); + } + + /// #1160 (requires ONNX model — ignored in CI, run locally): the + /// explanation for the default fusion strategy must carry vector ranks, + /// fusion contributions, and reproduce the plain recall scores. + #[test] + #[ignore = "requires ONNX embedder (model download) in CI"] + fn explain_fusion_reproduces_plain_recall() { + let (mut uteke, dir) = scratch(); + // Real embeddings (remember → model embed) so vector_similarity is a + // genuine cosine; synthetic fixtures would make it arbitrary. + uteke + .remember( + "quarterly revenue projections for the board meeting", + &["finance"], + None, + Some("explain-ns"), + ) + .unwrap(); + for i in 0..3 { + uteke + .remember( + &format!("unrelated filler note {i} about gardening"), + &["misc"], + None, + Some("explain-ns"), + ) + .unwrap(); + } + + let query = "quarterly revenue projections"; + + // Boosts disabled → explained must EXACTLY reproduce the plain call. + // (With boosts on, touch_access side effects legitimately make + // salience drift between separate pipeline runs.) + uteke.set_salience_recency_config(crate::salience_recency::SalienceRecencyConfig { + salience_weight: 0.0, + recency_weight: 0.0, + }); + + let plain = uteke + .recall_hybrid( + query, + 3, + None, + Some("explain-ns"), + RecallStrategy::Fusion, + 0.0, + ) + .unwrap(); + let explained = uteke + .recall_explained( + query, + 3, + None, + Some("explain-ns"), + RecallStrategy::Fusion, + 0.0, + ) + .unwrap(); + + assert!(!plain.is_empty()); + assert_eq!(plain.len(), explained.len()); + assert_eq!( + plain[0].memory.id, explained[0].result.memory.id, + "explained fusion must keep the same top-1 as plain fusion" + ); + assert!( + plain[0].memory.content.contains("quarterly revenue"), + "on-topic memory must rank first: {}", + plain[0].memory.content + ); + + for (p, e) in plain.iter().zip(explained.iter()) { + assert_eq!(p.memory.id, e.result.memory.id, "same ranking order"); + assert!( + (p.score - e.result.score).abs() < 1e-6, + "explained score must equal plain score: {} vs {}", + e.result.score, + p.score + ); + } + + let e = &explained[0].explanation; + assert_eq!(e.strategy, "fusion"); + assert!( + e.vector_similarity.unwrap_or(0.0) > 0.3, + "on-topic memory must have high vector similarity: {e:?}" + ); + assert_eq!(e.vector_rank, Some(1), "on-topic must be vector rank 1"); + let vc = e.fusion_vector_contribution.expect("vector contribution"); + assert!(vc > 0.0, "fusion contribution must be positive"); + // Contribution math: 1.7 / (60 + 1) for rank 1. + assert!( + (vc - 1.7 / 61.0).abs() < 1e-6, + "vector contribution for rank 1 must be 1.7/61: {vc}" + ); + + // ── Phase 2: default boosts on → explanation must be internally + // consistent: final == base + salience + recency. + uteke.set_salience_recency_config(crate::salience_recency::SalienceRecencyConfig { + salience_weight: 0.1, + recency_weight: 0.1, + }); + let explained_boosted = uteke + .recall_explained( + query, + 3, + None, + Some("explain-ns"), + RecallStrategy::Fusion, + 0.0, + ) + .unwrap(); + let eb = &explained_boosted[0].explanation; + let expected = + eb.base_score + eb.salience_boost.unwrap_or(0.0) + eb.recency_boost.unwrap_or(0.0); + assert!( + (eb.final_score - expected).abs() < 1e-4, + "final must equal base + boosts: {} vs {}", + eb.final_score, + expected + ); + let sal = eb.salience_boost.expect("salience delta"); + let rec = eb.recency_boost.expect("recency delta"); + assert!(sal > 0.0, "fresh important memory must gain salience"); + assert!( + (rec - 0.1).abs() < 0.02, + "brand-new memory must gain ~full 0.1 recency: {rec}" + ); + + drop(uteke); + drop(dir); + } +} diff --git a/crates/uteke-core/src/rooms.rs b/crates/uteke-core/src/rooms.rs index b30d9cb6..900ccf92 100644 --- a/crates/uteke-core/src/rooms.rs +++ b/crates/uteke-core/src/rooms.rs @@ -184,9 +184,10 @@ impl crate::Uteke { Ok(results) } - /// Delete a room and all its memory links. - /// Note: memories themselves are NOT deleted — they remain in their namespaces. - pub fn delete_room(&self, room_id: &str) -> Result<(), Error> { + /// Delete a room (unlink-only). + /// Returns the number of memory links removed; memories and documents + /// themselves are NOT deleted — they remain in their namespaces. + pub fn delete_room(&self, room_id: &str) -> Result { self.store.delete_room(room_id) } diff --git a/crates/uteke-core/src/timeline.rs b/crates/uteke-core/src/timeline.rs index d36ad948..1ef4deff 100644 --- a/crates/uteke-core/src/timeline.rs +++ b/crates/uteke-core/src/timeline.rs @@ -35,6 +35,11 @@ pub enum TimelineEventType { Tagged, /// Memory deleted. Forgot, + /// Superseded by a newer memory (#1172 Fase 2) — resolution recorded + /// with actor + evidence. + Superseded, + /// A supersession was undone (#1172 Fase 2) — memory restored. + SupersessionUndone, } impl TimelineEventType { @@ -46,6 +51,8 @@ impl TimelineEventType { Self::Consolidated => "consolidated", Self::Tagged => "tagged", Self::Forgot => "forgot", + Self::Superseded => "superseded", + Self::SupersessionUndone => "supersession_undone", } } @@ -57,6 +64,8 @@ impl TimelineEventType { "consolidated" => Some(Self::Consolidated), "tagged" => Some(Self::Tagged), "forgot" => Some(Self::Forgot), + "superseded" => Some(Self::Superseded), + "supersession_undone" => Some(Self::SupersessionUndone), _ => None, } } @@ -71,6 +80,15 @@ pub struct TimelineEvent { /// Optional JSON payload describing what changed. pub event_data: Option, pub created_at: String, + /// Who performed the event (#1172): agent id, "user", or "system". + /// `None` for events written before schema v18. + #[serde(default, skip_serializing_if = "Option::is_none")] + pub actor: Option, + /// JSON array of related memory IDs / scores supporting the event + /// (#1172) — e.g. contradiction-resolution evidence. `None` for events + /// written before schema v18. + #[serde(default, skip_serializing_if = "Option::is_none")] + pub evidence: Option, } impl Store { @@ -81,6 +99,23 @@ impl Store { memory_id: &str, event_type: TimelineEventType, event_data: Option<&serde_json::Value>, + ) -> Result<(), Error> { + self.add_timeline_event_with_provenance(memory_id, event_type, event_data, None, None) + } + + /// Append a timeline event with provenance attribution (#1172 Fase 1). + /// + /// `actor` = who performed the event (agent id, "user", "system"); + /// `evidence` = JSON array of related memory IDs / scores supporting the + /// event. Both optional; stored as `NULL` when absent. Best-effort like + /// [`Self::add_timeline_event`]. + pub fn add_timeline_event_with_provenance( + &self, + memory_id: &str, + event_type: TimelineEventType, + event_data: Option<&serde_json::Value>, + actor: Option<&str>, + evidence: Option<&serde_json::Value>, ) -> Result<(), Error> { let data_str = event_data.map(|v| match serde_json::to_string(v) { Ok(s) => s, @@ -91,15 +126,24 @@ impl Store { "null".to_string() } }); + let evidence_str = evidence.map(|v| match serde_json::to_string(v) { + Ok(s) => s, + Err(e) => { + tracing::warn!("timeline: failed to serialize event evidence: {e}"); + "null".to_string() + } + }); self.conn .execute( - "INSERT INTO timeline_events (memory_id, event_type, event_data, created_at) - VALUES (?1, ?2, ?3, ?4)", + "INSERT INTO timeline_events (memory_id, event_type, event_data, created_at, actor, evidence_json) + VALUES (?1, ?2, ?3, ?4, ?5, ?6)", params![ memory_id, event_type.as_str(), data_str, chrono::Utc::now().to_rfc3339(), + actor, + evidence_str, ], ) .map_err(|e| Error::db("add timeline event", e))?; @@ -115,10 +159,12 @@ impl Store { limit: usize, ) -> Result, Error> { let sql = if limit == 0 { - "SELECT id, memory_id, event_type, event_data, created_at FROM timeline_events + "SELECT id, memory_id, event_type, event_data, created_at, actor, evidence_json + FROM timeline_events WHERE memory_id = ?1 ORDER BY created_at DESC, id DESC" } else { - "SELECT id, memory_id, event_type, event_data, created_at FROM timeline_events + "SELECT id, memory_id, event_type, event_data, created_at, actor, evidence_json + FROM timeline_events WHERE memory_id = ?1 ORDER BY created_at DESC, id DESC LIMIT ?2" }; let mut stmt = self @@ -154,12 +200,25 @@ impl Store { } fn map_timeline_row(r: &rusqlite::Row) -> rusqlite::Result { + let evidence_str: Option = r.get(6)?; + let evidence = + evidence_str + .as_deref() + .and_then(|s| match serde_json::from_str::(s) { + Ok(v) => Some(v), + Err(e) => { + tracing::warn!("timeline: corrupted evidence JSON (will use null): {e}"); + None + } + }); Ok(TimelineEvent { id: r.get(0)?, memory_id: r.get(1)?, event_type: r.get(2)?, event_data: r.get(3)?, created_at: r.get(4)?, + actor: r.get(5)?, + evidence, }) } diff --git a/crates/uteke-mcp/Cargo.toml b/crates/uteke-mcp/Cargo.toml index 31a5b187..e58504f8 100644 --- a/crates/uteke-mcp/Cargo.toml +++ b/crates/uteke-mcp/Cargo.toml @@ -21,14 +21,15 @@ name = "uteke_mcp" path = "src/lib.rs" [features] -default = ["usearch"] -# Vector index backend forwarded to uteke-core. Exactly one must be enabled; -# default is usearch (HNSW C++ FFI). vecq = pure-Rust 4-bit quantization (#1098). +# Dual-engine default (#1168): ship BOTH engines, runtime-selectable via +# UTEKE_VECTOR_BACKEND / [vector] backend. Slim builds: --no-default-features +# --features vecq (mobile, pure Rust) or --features usearch (classic). +default = ["usearch", "vecq"] usearch = ["uteke-core/usearch"] vecq = ["uteke-core/vecq"] [dependencies] -uteke-core = { path = "../uteke-core", version = "0.16.0", default-features = false, features = ["onnx"] } +uteke-core = { path = "../uteke-core", version = "0.17.0", default-features = false, features = ["onnx"] } serde = { version = "1", features = ["derive"] } serde_json = "1" dirs = "6" diff --git a/crates/uteke-mcp/src/lib.rs b/crates/uteke-mcp/src/lib.rs index 6d0de145..92c41171 100644 --- a/crates/uteke-mcp/src/lib.rs +++ b/crates/uteke-mcp/src/lib.rs @@ -144,6 +144,9 @@ fn handle_request(uteke: &Uteke, method: &str, params: Option) -> Result< tool_search(), tool_list(), tool_get(), + tool_provenance(), + tool_contradictions(), + tool_contradictions_undo(), tool_supersede(), tool_update(), tool_forget(), @@ -175,6 +178,8 @@ fn handle_request(uteke: &Uteke, method: &str, params: Option) -> Result< tool_tags_list(), tool_tags_rename(), tool_tags_delete(), + tool_namespace_rename(), + tool_namespace_delete(), tool_pin(), tool_unpin(), ] @@ -194,6 +199,9 @@ fn handle_request(uteke: &Uteke, method: &str, params: Option) -> Result< "uteke_search" => exec_search(uteke, &arguments)?, "uteke_list" => exec_list(uteke, &arguments)?, "uteke_get" => exec_get(uteke, &arguments)?, + "uteke_provenance" => exec_provenance(uteke, &arguments)?, + "uteke_contradictions" => exec_contradictions(uteke, &arguments)?, + "uteke_contradictions_undo" => exec_contradictions_undo(uteke, &arguments)?, "uteke_supersede" => exec_supersede(uteke, &arguments)?, "uteke_update" => exec_update(uteke, &arguments)?, "uteke_forget" => exec_forget(uteke, &arguments)?, @@ -227,6 +235,8 @@ fn handle_request(uteke: &Uteke, method: &str, params: Option) -> Result< "uteke_tags_list" => exec_tags_list(uteke, &arguments)?, "uteke_tags_rename" => exec_tags_rename(uteke, &arguments)?, "uteke_tags_delete" => exec_tags_delete(uteke, &arguments)?, + "uteke_namespace_rename" => exec_namespace_rename(uteke, &arguments)?, + "uteke_namespace_delete" => exec_namespace_delete(uteke, &arguments)?, "uteke_pin" => exec_pin(uteke, &arguments)?, "uteke_unpin" => exec_unpin(uteke, &arguments)?, _ => return Err(format!("Unknown tool: {tool_name}")), @@ -275,7 +285,8 @@ fn tool_recall() -> Value { "tags": { "type": "array", "items": { "type": "string" }, "description": "Filter by tags (optional)" }, "min_score": { "type": "number", "description": "Minimum similarity score 0..1 (default: 0.0)" }, "type": { "type": "string", "enum": ["all", "memory", "doc"], "description": "Search type: 'all' (default, unified), 'memory', or 'doc'" }, - "strategy": { "type": "string", "enum": ["fusion", "hybrid", "vector", "fts5", "graph"], "description": "Recall strategy: 'fusion' (default since 0.16.0, weighted RRF of vector×1.7 + hybrid×1, #1123), 'hybrid' (vector+FTS5 via RRF), 'vector' (similarity only), 'fts5' (keyword only), or 'graph' (hybrid + graph-signal reranking)", "default": "fusion" } + "strategy": { "type": "string", "enum": ["fusion", "hybrid", "vector", "fts5", "graph"], "description": "Recall strategy: 'fusion' (default since 0.16.0, weighted RRF of vector×1.7 + hybrid×1, #1123), 'hybrid' (vector+FTS5 via RRF), 'vector' (similarity only), 'fts5' (keyword only), or 'graph' (hybrid + graph-signal reranking)", "default": "fusion" }, + "explain": { "type": "boolean", "description": "Return per-result ranking signals (#1160): vector similarity/rank, RRF contributions, jaccard/salience/recency/graph boosts. Memory-only — omitted type is treated as memory; explicit type=all/doc is rejected." } }, "required": ["query"] } @@ -312,6 +323,113 @@ fn tool_get() -> Value { }) } +fn tool_provenance() -> Value { + serde_json::json!({ + "name": "uteke_provenance", + "description": "Full provenance report for a memory (#1172): author/source fields, trust tier, source hash at write vs live content hash (tamper evidence), and the full timeline event chain with actor + evidence. Use when auditing why a memory exists, who wrote it, and whether it changed after write.", + "inputSchema": { + "type": "object", + "properties": { + "id": { "type": "string", "description": "Full UUID or unambiguous prefix" } + }, + "required": ["id"] + } + }) +} + +/// Full provenance report for a memory (#1172 Fase 1). +fn exec_provenance(uteke: &Uteke, args: &Value) -> Result { + let id_arg = args["id"].as_str().ok_or("Missing 'id'")?; + let id = resolve_id(uteke, id_arg)?; + + let report = uteke + .provenance(&id) + .map_err(|e| format!("Failed: {e}"))? + .ok_or_else(|| format!("Memory not found: {id}"))?; + + let text = serde_json::to_string_pretty(&report).unwrap_or_else(|_| "{}".to_string()); + Ok(ToolResult { + content: vec![McpContent::Text { + r#type: "text".to_string(), + text, + }], + is_error: false, + }) +} + +fn tool_contradictions() -> Value { + serde_json::json!({ + "name": "uteke_contradictions", + "description": "List the contradiction resolution ledger (#1172): memories that were superseded and soft-deprecated by conflict resolution, with winner, reason, and timestamp. Audit what the pipeline decided without digging through timeline events.", + "inputSchema": { + "type": "object", + "properties": { + "namespace": { "type": "string", "description": "Filter by namespace" }, + "limit": { "type": "integer", "description": "Max entries (default 50)" } + }, + "required": [] + } + }) +} + +fn tool_contradictions_undo() -> Value { + serde_json::json!({ + "name": "uteke_contradictions_undo", + "description": "Undo a contradiction resolution (#1172): restore a superseded memory to active, remove the supersession edge pair, and record the undo in the audit trail. Use when a conflict resolution was wrong.", + "inputSchema": { + "type": "object", + "properties": { + "id": { "type": "string", "description": "Full UUID or unambiguous prefix of the RETIRED memory to restore" } + }, + "required": ["id"] + } + }) +} + +/// Contradiction resolution ledger (#1172 Fase 2). +fn exec_contradictions(uteke: &Uteke, args: &Value) -> Result { + let namespace = args["namespace"].as_str(); + let limit = args["limit"].as_u64().unwrap_or(50) as usize; + + let resolutions = uteke + .contradiction_resolutions(namespace, limit) + .map_err(|e| format!("Failed: {e}"))?; + + let text = serde_json::to_string_pretty(&resolutions).unwrap_or_else(|_| "[]".to_string()); + Ok(ToolResult { + content: vec![McpContent::Text { + r#type: "text".to_string(), + text, + }], + is_error: false, + }) +} + +/// Undo a contradiction resolution (#1172 Fase 2). +fn exec_contradictions_undo(uteke: &Uteke, args: &Value) -> Result { + let id_arg = args["id"].as_str().ok_or("Missing 'id'")?; + let id = resolve_id(uteke, id_arg)?; + + match uteke + .undo_supersession(&id) + .map_err(|e| format!("Failed: {e}"))? + { + Some(winner) => { + let text = format!( + "\u{2713} Restored memory {id}\n was superseded by {winner}\n supersession edges removed \u{2014} the pair is no longer flagged" + ); + Ok(ToolResult { + content: vec![McpContent::Text { + r#type: "text".to_string(), + text, + }], + is_error: false, + }) + } + None => Err(format!("No supersession found for memory: {id}")), + } +} + fn tool_supersede() -> Value { serde_json::json!({ "name": "uteke_supersede", @@ -341,7 +459,8 @@ fn tool_update() -> Value { "metadata": { "type": "object", "description": "Replacement metadata JSON" }, "importance": { "type": "number", "description": "0.0–1.0" }, "pinned": { "type": "boolean", "description": "Pin (never decays) or unpin" }, - "memory_type": { "type": "string", "description": "fact | procedure | preference | decision | context | note | insight | reference | event" } + "memory_type": { "type": "string", "description": "fact | procedure | preference | decision | context | note | insight | reference | event" }, + "namespace": { "type": "string", "description": "Move the memory to this namespace (#1181 — plain move, no re-embed)" } }, "required": ["id"] } @@ -762,6 +881,37 @@ fn tool_tags_delete() -> Value { }) } +fn tool_namespace_rename() -> Value { + serde_json::json!({ + "name": "uteke_namespace_rename", + "description": "Rename a namespace, merging into the target when it already exists. Namespaces are a derived view — the old name vanishes when its last memory moved (#1181).", + "inputSchema": { + "type": "object", + "properties": { + "from": { "type": "string", "description": "Current namespace name" }, + "to": { "type": "string", "description": "New namespace name (existing name = merge)" } + }, + "required": ["from", "to"] + } + }) +} + +fn tool_namespace_delete() -> Value { + serde_json::json!({ + "name": "uteke_namespace_delete", + "description": "Delete a namespace with an explicit strategy for its memories: refuse (default — error while any memory uses the name), merge (move all memories to `target`), or deprecate (soft-delete — restorable via promote, never hard-deleted) (#1181).", + "inputSchema": { + "type": "object", + "properties": { + "name": { "type": "string", "description": "Namespace to delete" }, + "strategy": { "type": "string", "enum": ["refuse", "merge", "deprecate"], "description": "Default: refuse" }, + "target": { "type": "string", "description": "Target namespace when strategy = merge" } + }, + "required": ["name"] + } + }) +} + fn tool_pin() -> Value { serde_json::json!({ "name": "uteke_pin", @@ -900,6 +1050,40 @@ fn exec_recall(uteke: &Uteke, args: &Value) -> Result { None => uteke_core::RecallStrategy::Fusion, }; + // Explain mode (#1160): memory-only, bypasses unified results and + // returns full ranking signals per result. An omitted `type` (the + // documented default invocation) is accepted and treated as memory + // recall — only explicit all/doc are rejected (code-scanning fix: + // None maps to SearchType::All above, so the previous enum comparison + // wrongly rejected the default call). + if args["explain"].as_bool().unwrap_or(false) { + if !matches!(args["type"].as_str(), None | Some("memory")) { + return Err( + "explain is memory-only: pass \"type\": \"memory\" (or drop type)".to_string(), + ); + } + let explained = uteke + .recall_explained(query, limit, tags_ref, namespace, strategy, min_score) + .map_err(|e| format!("Failed: {e}"))?; + if explained.is_empty() { + return Ok(ToolResult { + content: vec![McpContent::Text { + r#type: "text".to_string(), + text: "No results found.".to_string(), + }], + is_error: false, + }); + } + let text = serde_json::to_string_pretty(&explained).unwrap_or_else(|_| "[]".to_string()); + return Ok(ToolResult { + content: vec![McpContent::Text { + r#type: "text".to_string(), + text, + }], + is_error: false, + }); + } + // Use unified search when type is specified or default (all). // Fall back to legacy recall only for backward compat with existing MCP consumers. let results = uteke @@ -1139,6 +1323,22 @@ fn exec_update(uteke: &Uteke, args: &Value) -> Result { }); } + // Namespace move (#1181) — plain column update, no re-embed needed. + if let Some(ns) = args["namespace"].as_str() { + let moved = uteke + .move_memory(&id, ns) + .map_err(|e| format!("Failed: {e}"))?; + if !moved { + return Ok(ToolResult { + content: vec![McpContent::Text { + r#type: "text".to_string(), + text: format!("Memory not found: {id}"), + }], + is_error: true, + }); + } + } + Ok(ToolResult { content: vec![McpContent::Text { r#type: "text".to_string(), @@ -1856,14 +2056,18 @@ fn exec_room_list(uteke: &Uteke, args: &Value) -> Result { fn exec_room_delete(uteke: &Uteke, args: &Value) -> Result { let room_id = args["room_id"].as_str().ok_or("Missing 'room_id'")?; - uteke + let unlinked = uteke .delete_room(room_id) .map_err(|e| format!("Failed to delete room: {e}"))?; + let text = format!( + "Room '{room_id}' deleted. {unlinked} memory link(s) removed; the memories themselves are preserved in their namespaces (no longer linked to any room)." + ); + Ok(ToolResult { content: vec![McpContent::Text { r#type: "text".to_string(), - text: format!("Room deleted: {room_id}"), + text, }], is_error: false, }) @@ -2212,6 +2416,64 @@ fn exec_tags_delete(uteke: &Uteke, args: &Value) -> Result { }) } +/// Rename a namespace, merging into an existing target (#1181). +fn exec_namespace_rename(uteke: &Uteke, args: &Value) -> Result { + let from = args["from"].as_str().ok_or("Missing 'from'")?; + let to = args["to"].as_str().ok_or("Missing 'to'")?; + + let result = uteke + .rename_namespace(from, to) + .map_err(|e| format!("Failed: {e}"))?; + + let kind = if result.target_existed { + "merged into existing" + } else { + "renamed to" + }; + Ok(ToolResult { + content: vec![McpContent::Text { + r#type: "text".to_string(), + text: format!( + "✓ Namespace '{}' {} '{}' — {} memories moved", + result.from, kind, result.to, result.moved + ), + }], + is_error: false, + }) +} + +/// Delete a namespace with an explicit memory-fate strategy (#1181). +fn exec_namespace_delete(uteke: &Uteke, args: &Value) -> Result { + let name = args["name"].as_str().ok_or("Missing 'name'")?; + let strategy = args["strategy"].as_str().unwrap_or("refuse"); + let target = args["target"].as_str(); + + let result = uteke + .delete_namespace(name, strategy, target) + .map_err(|e| format!("Failed: {e}"))?; + + let text = match result.strategy.as_str() { + "merge" => format!( + "✓ Moved {} memories from '{}' to '{}' — namespace removed", + result.affected, + result.name, + result.target.as_deref().unwrap_or("?") + ), + "deprecate" => format!( + "✓ Soft-deleted {} memories in '{}' (restorable via promote; the name stays visible as deprecated-only)", + result.affected, result.name + ), + other => format!("✓ Namespace '{}' deleted (strategy={other})", result.name), + }; + Ok(ToolResult { + content: vec![McpContent::Text { + r#type: "text".to_string(), + text, + }], + is_error: false, + }) +} + fn exec_pin(uteke: &Uteke, args: &Value) -> Result { let id_arg = args["id"].as_str().ok_or("Missing 'id'")?; let id = resolve_id(uteke, id_arg)?; @@ -2487,3 +2749,97 @@ mod supersession_mcp_tests { std::fs::remove_dir_all(&dir).ok(); } } + +#[cfg(test)] +mod contradiction_mcp_tests { + use super::*; + use uteke_core::Uteke; + + #[test] + fn exec_contradictions_list_and_undo_roundtrip() { + let dir = std::env::temp_dir().join(format!( + "scmcp-{}-{}", + std::process::id(), + std::time::SystemTime::now() + .duration_since(std::time::UNIX_EPOCH) + .map(|d| d.as_nanos()) + .unwrap_or(0) + )); + std::fs::create_dir_all(&dir).unwrap(); + let uteke = Uteke::open(dir.join("t.db").to_str().unwrap()).unwrap(); + + let mk = |content: &str| -> String { + use uteke_core::memory::types::Memory; + let id = uuid::Uuid::new_v4().to_string(); + let m = Memory { + id: id.clone(), + content: content.to_string(), + embedding: vec![0.5; 768], + tags: vec![], + metadata: serde_json::json!({}), + created_at: chrono::Utc::now(), + updated_at: chrono::Utc::now(), + namespace: "scmcp-ns".to_string(), + access_count: 0, + last_accessed: None, + deprecated: false, + deprecated_at: None, + valid_from: None, + valid_until: None, + memory_type: "decision".to_string(), + importance: 0.5, + pinned: false, + content_type: "text".to_string(), + slug: None, + source: None, + source_type: "user".to_string(), + author_type: "agent".to_string(), + }; + uteke.store().insert(&m).unwrap(); + id + }; + let old = mk("old decision"); + let new = mk("new decision"); + uteke.supersede(&old, &new, Some("pivot")).unwrap(); + + // Ledger lists the superseded memory. + let result = exec_contradictions( + &uteke, + &serde_json::json!({"namespace": "scmcp-ns", "limit": 50}), + ) + .unwrap(); + assert!(!result.is_error); + let text = match &result.content[0] { + McpContent::Text { text, .. } => text.clone(), + #[allow(unreachable_patterns)] + _ => panic!("expected text content"), + }; + let ledger: serde_json::Value = serde_json::from_str(&text).unwrap(); + let entries = ledger.as_array().unwrap(); + assert_eq!(entries.len(), 1, "one entry expected: {text}"); + assert_eq!(entries[0]["id"], serde_json::json!(old)); + + // Undo via short id (resolution contract). + let old_short: String = old.chars().take(8).collect(); + let undone = + exec_contradictions_undo(&uteke, &serde_json::json!({"id": old_short})).unwrap(); + assert!(!undone.is_error); + + // Ledger empty after undo; the memory is active again. + let result = exec_contradictions(&uteke, &serde_json::json!({})).unwrap(); + let text = match &result.content[0] { + McpContent::Text { text, .. } => text.clone(), + #[allow(unreachable_patterns)] + _ => panic!("expected text content"), + }; + assert_eq!(text.trim(), "[]", "ledger must be empty after undo: {text}"); + assert!(!uteke.get_by_id(&old).unwrap().unwrap().deprecated); + + // Undo again → loud error (no supersession left). + let err = exec_contradictions_undo(&uteke, &serde_json::json!({"id": &old})); + assert!(err.is_err(), "second undo must error loudly"); + + drop(uteke); + std::fs::remove_dir_all(&dir).ok(); + } +} diff --git a/crates/uteke-server/Cargo.toml b/crates/uteke-server/Cargo.toml index 044d5bb6..18864105 100644 --- a/crates/uteke-server/Cargo.toml +++ b/crates/uteke-server/Cargo.toml @@ -10,10 +10,11 @@ keywords = ["server", "memory", "ai", "http", "embedding"] categories = ["web-programming::http-server", "development-tools"] [features] -default = ["usearch"] +# Dual-engine default (#1168): ship BOTH engines, runtime-selectable via +# UTEKE_VECTOR_BACKEND / [vector] backend. Slim builds: --no-default-features +# --features vecq (mobile, pure Rust) or --features usearch (classic). +default = ["usearch", "vecq"] docgen = ["dep:schemars"] -# Vector index backend forwarded to uteke-core. Exactly one must be enabled; -# default is usearch (HNSW, C++ FFI). vecq = pure-Rust 4-bit quantization (#1098). usearch = ["uteke-core/usearch", "uteke-mcp/usearch"] vecq = ["uteke-core/vecq", "uteke-mcp/vecq"] @@ -26,8 +27,8 @@ name = "uteke-serve" path = "src/main.rs" [dependencies] -uteke-core = { path = "../uteke-core", version = "0.16.0", default-features = false, features = ["onnx"] } -uteke-mcp = { path = "../uteke-mcp", version = "0.16.0", default-features = false } +uteke-core = { path = "../uteke-core", version = "0.17.0", default-features = false, features = ["onnx"] } +uteke-mcp = { path = "../uteke-mcp", version = "0.17.0", default-features = false } serde = { version = "1", features = ["derive"] } serde_json = "1" schemars = { version = "1", optional = true } diff --git a/crates/uteke-server/src/api_registry.rs b/crates/uteke-server/src/api_registry.rs index 2439cd98..8ded085f 100644 --- a/crates/uteke-server/src/api_registry.rs +++ b/crates/uteke-server/src/api_registry.rs @@ -58,11 +58,29 @@ pub const ENDPOINTS: &[Endpoint] = &[ Endpoint { method: "GET", path: "/namespaces", - description: "List all namespaces in the memory store", + description: "List all namespaces in the memory store. `?with_counts=true` adds `count` (total) plus `active`/`deprecated` breakdown fields (#1181).", request_type: None, response_type: None, excludes_deprecated: false, - issues: &[], + issues: &["1181"], + }, + Endpoint { + method: "POST", + path: "/namespaces/rename", + description: "Rename a namespace (`{from, to}`). When `to` already exists this is a merge; the old name vanishes (derived view). Returns `{from, to, moved, target_existed}` (#1181).", + request_type: Some("NamespaceRenameRequest"), + response_type: None, + excludes_deprecated: false, + issues: &["1181"], + }, + Endpoint { + method: "POST", + path: "/namespaces/delete", + description: "Delete a namespace with an explicit strategy for its memories (`{name, strategy, target?}`): `refuse` (default — 409 while any memory references the name), `merge` (move all memories to `target`), or `deprecate` (soft-delete — restorable, never hard-deleted) (#1181).", + request_type: Some("NamespaceDeleteRequest"), + response_type: None, + excludes_deprecated: false, + issues: &["1181"], }, Endpoint { method: "GET", @@ -95,7 +113,7 @@ pub const ENDPOINTS: &[Endpoint] = &[ Endpoint { method: "POST", path: "/recall", - description: "Semantic search — recall memories by meaning. Returns ranked results.", + description: "Semantic search — recall memories by meaning. Returns ranked results. Set `explain: true` (#1160) to include per-result ranking signals (vector similarity/rank, RRF contributions, boosts); memory-only recall.", request_type: Some("RecallRequest"), response_type: None, excludes_deprecated: true, @@ -113,7 +131,7 @@ pub const ENDPOINTS: &[Endpoint] = &[ Endpoint { method: "POST", path: "/list", - description: "List memories with optional filters (namespace, tags, sort, limit, offset).", + description: "List memories with optional filters (namespace, tags, sort, limit, offset). Set `include_meta: true` (#1188) for a pagination envelope `{memories, total, has_more, next_offset}` instead of the bare array (not supported with `at`).", request_type: Some("ListParams"), response_type: None, excludes_deprecated: true, @@ -314,7 +332,7 @@ pub const ENDPOINTS: &[Endpoint] = &[ Endpoint { method: "DELETE", path: "/room/delete", - description: "Delete a room and all its memories. Accepts `?room_id=...` query param.", + description: "Delete a room (unlink-only): room links are removed, memories and documents are preserved. Response: `{ deleted, unlinked_memories }`. Accepts `?room_id=...` query param.", request_type: None, response_type: None, excludes_deprecated: false, @@ -452,20 +470,20 @@ pub const ENDPOINTS: &[Endpoint] = &[ Endpoint { method: "POST", path: "/graph/edge", - description: "Add a directed edge between two memories.", + description: "Add a directed edge between two memories. Accepts memory IDs (a linked graph node is ensured automatically, #1180) or existing graph node IDs. Returns `{ok, source_node, target_node}`.", request_type: Some("GraphEdgeRequest"), response_type: None, excludes_deprecated: false, - issues: &[], + issues: &["1180"], }, Endpoint { method: "DELETE", path: "/graph/edge", - description: "Remove an edge between two memories. Accepts `?from=...&to=...` query params.", + description: "Remove an edge between two nodes. Accepts memory IDs or graph node IDs via `?source=...&target=...` query params (#1180).", request_type: None, response_type: None, excludes_deprecated: false, - issues: &[], + issues: &["1180"], }, Endpoint { method: "GET", @@ -477,6 +495,33 @@ pub const ENDPOINTS: &[Endpoint] = &[ issues: &[], }, // ── Timeline ───────────────────────────────────────────────────────── + Endpoint { + method: "GET", + path: "/provenance", + description: "Full provenance report for a memory (#1172): author/source fields, trust tier, source hash at write vs live-recomputed content hash (tamper evidence), and the full timeline event chain with actor + evidence. Accepts `?id=...` query param.", + request_type: None, + response_type: None, + excludes_deprecated: false, + issues: &["1172"], + }, + Endpoint { + method: "GET", + path: "/contradictions", + description: "List superseded-but-not-restored memories (#1172) — the auditable contradiction resolution ledger (deprecated rows carrying a live superseded_by edge). Accepts `?namespace=...&limit=...`.", + request_type: None, + response_type: None, + excludes_deprecated: false, + issues: &["1172"], + }, + Endpoint { + method: "POST", + path: "/contradictions/undo", + description: "Undo a contradiction resolution (#1172): restore the retired memory to active, remove the supersession pair, and record supersession_undone provenance events. Body: `{id}`.", + request_type: Some("ContradictionUndoRequest"), + response_type: None, + excludes_deprecated: false, + issues: &["1172"], + }, Endpoint { method: "GET", path: "/timeline", diff --git a/crates/uteke-server/src/handlers.rs b/crates/uteke-server/src/handlers.rs index df7050a9..6b52b308 100644 --- a/crates/uteke-server/src/handlers.rs +++ b/crates/uteke-server/src/handlers.rs @@ -370,6 +370,47 @@ pub fn route(uteke: &Mutex, ctx: &ReqCtx, req: &mut Request) -> Response< None => uteke_core::RecallStrategy::Fusion, }; + // Explain mode (#1160): memory-only recall with per-result + // ranking signals. Mutually exclusive with the unified / + // time-travel / temporal surfaces. + if req_data.explain { + if req_data.search_type.is_some() { + return ctx.error_response_for( + req, + 400, + "explain is memory-only: omit search_type", + ); + } + if point_in_time.is_some() || has_temporal { + return ctx.error_response_for( + req, + 400, + "explain cannot be combined with at/before/after", + ); + } + if entity_filter.is_some() || category_filter.is_some() || req_data.enrich { + return ctx.error_response_for( + req, + 400, + "explain cannot be combined with entity/category/enrich", + ); + } + return match uteke.recall_explained( + &req_data.query, + limit, + tags_filter, + ns(&req_data.namespace), + strategy, + min_score, + ) { + Ok(explained) => ctx.ok_response_for(req, &explained), + Err(e) => { + error!("Explain recall error: {e}"); + ctx.error_response_for(req, 500, "Internal server error") + } + }; + } + // Unified search path (#531): when search_type is specified, // use recall_unified. Entity/category filters are passed // through to the core recall candidate loop (#663). @@ -582,7 +623,35 @@ pub fn route(uteke: &Mutex, ctx: &ReqCtx, req: &mut Request) -> Response< ), }; match list_result { - Ok(memories) => ctx.ok_response_for(req, &memories), + Ok(memories) => { + // #1188: opt-in pagination envelope. The default + // response stays a bare array for existing clients. + if req_data.include_meta && req_data.at.is_none() { + // Total matching rows (same filters as the page). + let total = match &req_data.tag { + Some(tag) => uteke.count_by_tag(tag, ns(&req_data.namespace)), + None => uteke.count(ns(&req_data.namespace)), + } + .unwrap_or(0); + let fetched = memories.len(); + let has_more = req_data.offset + fetched < total; + let next_offset = if has_more { + Some(req_data.offset + fetched) + } else { + None + }; + return ctx.ok_response_for( + req, + &serde_json::json!({ + "memories": memories, + "total": total, + "has_more": has_more, + "next_offset": next_offset, + }), + ); + } + ctx.ok_response_for(req, &memories) + } Err(e) => { error!("Internal error: {e}"); ctx.error_response_for(req, 500, "Internal server error") @@ -811,16 +880,26 @@ pub fn route(uteke: &Mutex, ctx: &ReqCtx, req: &mut Request) -> Response< (Method::Get, "/namespaces") => { let with_counts = path.contains("with_counts=true"); if with_counts { - match uteke.list_namespaces_with_counts() { + match uteke.list_namespaces_with_lifecycle_counts() { Ok(counts) => { #[derive(serde::Serialize)] struct NamespaceCount { name: String, count: usize, + /// Memories not deprecated (#1181). + active: usize, + /// Deprecated memories — additive fields, `count` keeps + /// the total so existing clients stay correct (#1181). + deprecated: usize, } let result: Vec = counts .into_iter() - .map(|(name, count)| NamespaceCount { name, count }) + .map(|(name, active, deprecated)| NamespaceCount { + count: active + deprecated, + name, + active, + deprecated, + }) .collect(); ctx.ok_response_for(req, &result) } @@ -840,6 +919,53 @@ pub fn route(uteke: &Mutex, ctx: &ReqCtx, req: &mut Request) -> Response< } } + // ── Namespace Rename/Merge (#1181) ────────────────────────────── + (Method::Post, "/namespaces/rename") => { + match read_body::(req.as_reader()) { + Ok(req_data) => match uteke.rename_namespace(&req_data.from, &req_data.to) { + Ok(result) => ctx.ok_response_for(req, &result), + Err(uteke_core::Error::Validation(msg)) => { + ctx.error_response_for(req, 400, msg) + } + Err(e) => { + error!("Namespace rename error: {e}"); + ctx.error_response_for(req, 500, "Internal server error") + } + }, + Err(e) => ctx.error_response_for(req, 400, e), + } + } + + // ── Namespace Delete with explicit strategy (#1181) ───────────── + (Method::Post, "/namespaces/delete") => { + match read_body::(req.as_reader()) { + Ok(req_data) => { + match uteke.delete_namespace( + &req_data.name, + &req_data.strategy, + req_data.target.as_deref(), + ) { + Ok(result) => ctx.ok_response_for(req, &result), + Err(uteke_core::Error::Validation(msg)) => { + // `refuse` on a non-empty namespace → 409 Conflict; + // other validation failures → 400. + let status = if req_data.strategy == "refuse" { + 409 + } else { + 400 + }; + ctx.error_response_for(req, status, msg) + } + Err(e) => { + error!("Namespace delete error: {e}"); + ctx.error_response_for(req, 500, "Internal server error") + } + } + } + Err(e) => ctx.error_response_for(req, 400, e), + } + } + // ── Recent (#528) ────────────────────────────────────────────── (Method::Get, "/recent") => { let ns = parse_query_namespace(&path); @@ -883,43 +1009,74 @@ pub fn route(uteke: &Mutex, ctx: &ReqCtx, req: &mut Request) -> Response< ); } - // Validate both nodes exist as memories (issue #542 acceptance criteria) - match uteke.get_by_id(&req_data.source) { - Ok(Some(_)) => {} - Ok(None) => { - return ctx.error_response_for( - req, - 404, - format!("Source memory not found: {}", req_data.source), - ); - } - Err(e) => { - error!("Internal error: {e}"); - return ctx.error_response_for(req, 500, "Internal server error"); + let conn = uteke.graph_store(); + let gs = uteke_core::graph::GraphStore::new(conn); + + // Resolve an input ID to its graph node without creating + // anything: memory-linked nodes first, then explicit node IDs + // (clients may pass IDs from GET /graph) (#1180). + let resolve_node = |id: &str| -> Result, uteke_core::Error> { + if let Some(nid) = gs.node_id_for_memory(id)? { + return Ok(Some(nid)); } - } - match uteke.get_by_id(&req_data.target) { - Ok(Some(_)) => {} - Ok(None) => { - return ctx.error_response_for( - req, - 404, - format!("Target memory not found: {}", req_data.target), - ); + gs.get_node(id).map(|n| n.map(|node| node.id)) + }; + + // Validate both sides exist as memories or graph nodes — + // memory IDs are ensured into nodes below (#1180, relaxes the + // memory-only #542 check that made every valid call 500). + for (role, id) in [("Source", &req_data.source), ("Target", &req_data.target)] { + if matches!(uteke.get_by_id(id), Ok(Some(_))) { + continue; } - Err(e) => { - error!("Internal error: {e}"); - return ctx.error_response_for(req, 500, "Internal server error"); + match resolve_node(id) { + Ok(Some(_)) => {} + Ok(None) => { + return ctx.error_response_for( + req, + 404, + format!("{role} memory not found: {id}"), + ); + } + Err(e) => { + error!("Graph resolve error: {e}"); + return ctx.error_response_for(req, 500, "Internal server error"); + } } } - let conn = uteke.graph_store(); - let gs = uteke_core::graph::GraphStore::new(conn); let relation = req_data.edge_type.as_deref().unwrap_or("related"); let weight = req_data.weight.unwrap_or(1.0); - match gs.add_edge(&req_data.source, &req_data.target, relation, weight) { - Ok(()) => ctx.ok_response_for(req, &serde_json::json!({"ok": true})), + // Map memory IDs → graph node IDs before insertion (#1180). + // `graph_edges.source_id/target_id` carry FKs to + // `graph_nodes(id)`, so inserting raw memory IDs violates the + // FK and always 500s. Validation above guarantees each ID is + // a memory (node gets ensured) or an existing node. + let ensure = |id: &str| -> Result { + match resolve_node(id)? { + Some(nid) => Ok(nid), + None => gs.ensure_node_for_memory(id), + } + }; + let (src_node, tgt_node) = + match (ensure(&req_data.source), ensure(&req_data.target)) { + (Ok(s), Ok(t)) => (s, t), + (Err(e), _) | (_, Err(e)) => { + error!("Graph ensure_node error: {e}"); + return ctx.error_response_for(req, 500, "Internal server error"); + } + }; + + match gs.add_edge(&src_node, &tgt_node, relation, weight) { + Ok(()) => ctx.ok_response_for( + req, + &serde_json::json!({ + "ok": true, + "source_node": src_node, + "target_node": tgt_node, + }), + ), Err(e) => { error!("Graph add_edge error: {e}"); ctx.error_response_for(req, 500, "Internal server error") @@ -939,7 +1096,29 @@ pub fn route(uteke: &Mutex, ctx: &ReqCtx, req: &mut Request) -> Response< (Some(src), Some(tgt)) => { let conn = uteke.graph_store(); let gs = uteke_core::graph::GraphStore::new(conn); - match gs.remove_edge(src, tgt) { + // Accept either memory IDs or graph node IDs (#1180) — + // resolve both sides to node IDs before deleting. + let resolve = |id: &str| -> Result, uteke_core::Error> { + if let Some(nid) = gs.node_id_for_memory(id)? { + return Ok(Some(nid)); + } + gs.get_node(id).map(|n| n.map(|node| node.id)) + }; + let (src_node, tgt_node) = match (resolve(src), resolve(tgt)) { + (Ok(Some(s)), Ok(Some(t))) => (s, t), + (Ok(_), Ok(_)) => { + return ctx.error_response_for( + req, + 404, + format!("Edge not found: {src} -> {tgt}"), + ); + } + (Err(e), _) | (_, Err(e)) => { + error!("Graph node resolve error: {e}"); + return ctx.error_response_for(req, 500, "Internal server error"); + } + }; + match gs.remove_edge(&src_node, &tgt_node) { Ok(true) => ctx.ok_response_for(req, &serde_json::json!({"ok": true})), Ok(false) => ctx.error_response_for( req, @@ -1265,11 +1444,12 @@ pub fn route(uteke: &Mutex, ctx: &ReqCtx, req: &mut Request) -> Response< && req_data.importance.is_none() && req_data.pinned.is_none() && req_data.memory_type.is_none() + && req_data.namespace.is_none() { return ctx.error_response_for( req, 400, - "No fields to update. Provide at least one of: content, tags, metadata, importance, pinned, memory_type", + "No fields to update. Provide at least one of: content, tags, metadata, importance, pinned, memory_type, namespace", ); } let tag_refs: Option> = req_data.tags; @@ -1284,6 +1464,17 @@ pub fn route(uteke: &Mutex, ctx: &ReqCtx, req: &mut Request) -> Response< req_data.memory_type.as_deref(), ) { Ok(true) => { + // Namespace move (#1181): separate op after the field + // update — plain column change, no re-embed needed. + if let Some(ns) = req_data.namespace.as_deref() { + if let Err(e) = uteke.move_memory(&req_data.id, ns) { + let status = match e { + uteke_core::Error::Validation(_) => 400, + _ => 500, + }; + return ctx.error_response_for(req, status, e.to_string()); + } + } ctx.ok_response_for(req, &serde_json::json!({"updated": req_data.id})) } Ok(false) => ctx.error_response_for( @@ -1641,7 +1832,14 @@ pub fn route(uteke: &Mutex, ctx: &ReqCtx, req: &mut Request) -> Response< } }; match uteke.delete_room(&room_id) { - Ok(()) => ctx.ok_response_for(req, &serde_json::json!({"deleted": room_id})), + Ok(unlinked) => ctx.ok_response_for( + req, + &serde_json::json!({ + "deleted": room_id, + "unlinked_memories": unlinked, + "note": "memories and documents are preserved in their namespaces, no longer linked to any room" + }), + ), Err(e) => { let msg = format!("{e}"); if msg.contains("not found") { @@ -2056,6 +2254,70 @@ pub fn route(uteke: &Mutex, ctx: &ReqCtx, req: &mut Request) -> Response< Err(e) => ctx.error_response_for(req, 400, e), }, + // ── Provenance chain (#1172) ───────────────────────────────────── + (Method::Get, "/provenance") => { + let query = path.split('?').nth(1).unwrap_or(""); + let id = match parse_query_param(query, "id") { + Some(id) => id, + None => return ctx.error_response_for(req, 400, "Missing 'id' query parameter"), + }; + match uteke.provenance(&id) { + Ok(Some(report)) => ctx.ok_response_for(req, &report), + Ok(None) => ctx.error_response_for(req, 404, format!("Memory not found: {id}")), + Err(e) => { + error!("Provenance error: {e}"); + ctx.error_response_for(req, 500, "Internal server error") + } + } + } + + // ── Contradiction resolutions ledger (#1172) ──────────────────── + (Method::Get, "/contradictions") => { + let query = path.split('?').nth(1).unwrap_or(""); + let ns = parse_query_namespace(&path); + let limit = parse_query_param(query, "limit") + .and_then(|v| v.parse::().ok()) + .unwrap_or(50); + match uteke.contradiction_resolutions(ns.as_deref(), limit) { + Ok(resolutions) => ctx.ok_response_for(req, &resolutions), + Err(e) => { + error!("Contradiction resolutions error: {e}"); + ctx.error_response_for(req, 500, "Internal server error") + } + } + } + + // ── Undo a contradiction resolution (#1172) ───────────────────── + (Method::Post, "/contradictions/undo") => { + #[derive(Deserialize)] + struct ContradictionUndoRequest { + /// ID of the retired (superseded) memory to restore. + id: String, + } + match read_body::(req.as_reader()) { + Ok(req_data) => match uteke.undo_supersession(&req_data.id) { + Ok(Some(winner)) => ctx.ok_response_for( + req, + &serde_json::json!({ + "undone": true, + "restored": req_data.id, + "was_superseded_by": winner, + }), + ), + Ok(None) => ctx.error_response_for( + req, + 404, + format!("No supersession found for memory: {}", req_data.id), + ), + Err(e) => { + error!("Contradiction undo error: {e}"); + ctx.error_response_for(req, 500, "Internal server error") + } + }, + Err(e) => ctx.error_response_for(req, 400, e), + } + } + // ── Timeline ───────────────────────────────────────────────────── (Method::Get, "/timeline") => { let query = path.split('?').nth(1).unwrap_or(""); @@ -2578,4 +2840,803 @@ mod room_recall_at_tests { // missing id_remove → 400 at deserialize assert!(serde_json::from_str::(r#"{"id_keep":"abc"}"#).is_err()); } + + // ── /graph/edge memory-ID contract (#1180) ───────────────────────── + + use tiny_http::TestRequest; + + struct GraphEdgeApp { + uteke: Mutex, + } + + impl GraphEdgeApp { + fn new() -> Self { + // No embedder: these endpoints are storage-only, and tests must + // run in CI builds without the ONNX runtime lib. + Self { + uteke: Mutex::new( + Uteke::open_with_backend(":memory:", None) + .expect("open in-memory uteke without embedder"), + ), + } + } + + fn call( + &self, + method: Method, + url: &str, + body: Option, + ) -> (u16, serde_json::Value) { + let mut req = match body { + Some(b) => { + let leaked: &'static str = Box::leak(b.into_boxed_str()); + TestRequest::new() + .with_method(method) + .with_path(url) + .with_body(leaked) + .into() + } + None => TestRequest::new().with_method(method).with_path(url).into(), + }; + let ctx = ReqCtx { + auth_token_hash: None, + read_only_token_hash: None, + cors_origins: Vec::new(), + recall_config: None, + extraction_config: None, + }; + let resp = route(&self.uteke, &ctx, &mut req); + let status = resp.status_code().0; + let bytes = resp.into_reader().into_inner(); + let json = serde_json::from_slice(&bytes).unwrap_or(serde_json::Value::Null); + (status, json) + } + + fn remember(&self, content: &str) -> String { + let body = serde_json::json!({ "content": content }).to_string(); + let (status, resp) = self.call(Method::Post, "/remember", Some(body)); + assert_eq!(status, 200, "remember must succeed: {resp}"); + resp["id"] + .as_str() + .unwrap_or_else(|| panic!("remember response must carry id: {resp}")) + .to_string() + } + + fn add_edge(&self, source: &str, target: &str) -> (u16, serde_json::Value) { + let body = serde_json::json!({ "source": source, "target": target }).to_string(); + self.call(Method::Post, "/graph/edge", Some(body)) + } + + fn delete_edge(&self, source: &str, target: &str) -> (u16, serde_json::Value) { + let url = format!("/graph/edge?source={source}&target={target}"); + self.call(Method::Delete, &url, None) + } + } + + #[test] + fn graph_edge_post_with_memory_ids_returns_ok() { + let app = GraphEdgeApp::new(); + let id1 = app.remember("Uteke is a local-first memory engine"); + let id2 = app.remember("Corin renders the memory graph"); + + let (status, resp) = app.add_edge(&id1, &id2); + assert_eq!( + status, 200, + "valid memory IDs must not 500 (FK mismatch #1180): {resp}" + ); + assert_eq!(resp["ok"], serde_json::json!(true)); + } + + #[test] + fn graph_edge_post_resolves_memory_ids_to_nodes() { + let app = GraphEdgeApp::new(); + let id1 = app.remember("alpha memory"); + let id2 = app.remember("beta memory"); + + let (status, resp) = app.add_edge(&id1, &id2); + assert_eq!(status, 200, "{resp}"); + + let src_node = resp["source_node"].as_str().expect("source_node id"); + let tgt_node = resp["target_node"].as_str().expect("target_node id"); + + // The resolved nodes must exist in GET /graph and carry the memory + // link, so visualization can map edges back to memories. + let (_, graph) = app.call(Method::Get, "/graph", None); + let nodes = graph["nodes"].as_array().expect("nodes array"); + assert!( + nodes.iter().any(|n| n["id"] == serde_json::json!(src_node) + && n["memory_id"] == serde_json::json!(id1)), + "source node must link back to memory {id1}: {graph}" + ); + assert!( + nodes.iter().any(|n| n["id"] == serde_json::json!(tgt_node) + && n["memory_id"] == serde_json::json!(id2)), + "target node must link back to memory {id2}: {graph}" + ); + + // The edge itself must be visible in the graph payload. + let edges = graph["edges"].as_array().expect("edges array"); + assert!( + edges + .iter() + .any(|e| e["source_id"] == serde_json::json!(src_node) + && e["target_id"] == serde_json::json!(tgt_node) + && e["relation"] == serde_json::json!("related")), + "edge must appear in GET /graph: {graph}" + ); + } + + #[test] + fn graph_edge_post_accepts_graph_node_ids_too() { + let app = GraphEdgeApp::new(); + let id1 = app.remember("source memory"); + let id2 = app.remember("target memory"); + + let (_, first) = app.add_edge(&id1, &id2); + let src_node = first["source_node"].as_str().unwrap().to_string(); + + // A client holding IDs from GET /graph must still be able to link a + // graph node directly (e.g. entity nodes without a memory link). + let (status, resp) = app.add_edge(&src_node, &id2); + assert_eq!(status, 200, "node IDs must stay accepted: {resp}"); + } + + #[test] + fn graph_edge_post_with_unknown_memory_returns_404() { + let app = GraphEdgeApp::new(); + let ghost1 = "00000000-0000-4000-8000-000000000000"; + let ghost2 = "11111111-1111-4111-8111-111111111111"; + + let (status, resp) = app.add_edge(ghost1, ghost2); + assert_eq!(status, 404, "unknown source must 404: {resp}"); + } + + #[test] + fn graph_edge_post_self_loop_rejected() { + let app = GraphEdgeApp::new(); + let id1 = app.remember("self loop probe"); + + let (status, _) = app.add_edge(&id1, &id1); + assert_eq!(status, 400, "self-loop must stay rejected"); + } + + #[test] + fn graph_edge_delete_accepts_memory_ids() { + let app = GraphEdgeApp::new(); + let id1 = app.remember("edge delete probe a"); + let id2 = app.remember("edge delete probe b"); + + let (status, resp) = app.add_edge(&id1, &id2); + assert_eq!(status, 200, "{resp}"); + + // DELETE with the same memory IDs must resolve back to the nodes. + let (status, resp) = app.delete_edge(&id1, &id2); + assert_eq!(status, 200, "delete by memory IDs: {resp}"); + assert_eq!(resp["ok"], serde_json::json!(true)); + + // Second delete: edge gone → 404, not 500. + let (status, resp) = app.delete_edge(&id1, &id2); + assert_eq!(status, 404, "deleting a removed edge must 404: {resp}"); + } + + #[test] + fn graph_edge_delete_accepts_node_ids_and_unknown_ids() { + let app = GraphEdgeApp::new(); + let id1 = app.remember("node id delete probe a"); + let id2 = app.remember("node id delete probe b"); + + let (_, resp) = app.add_edge(&id1, &id2); + let src_node = resp["source_node"].as_str().unwrap().to_string(); + let tgt_node = resp["target_node"].as_str().unwrap().to_string(); + + // Node IDs (from GET /graph payloads) must keep working. + let (status, resp) = app.delete_edge(&src_node, &tgt_node); + assert_eq!(status, 200, "delete by node IDs: {resp}"); + + // Fully unknown IDs must 404 — never 500. + let (status, resp) = app.delete_edge("ghost-a", "ghost-b"); + assert_eq!(status, 404, "unknown IDs must 404: {resp}"); + } + + #[test] + fn graph_edge_delete_requires_both_params() { + let app = GraphEdgeApp::new(); + // URL built separately so the api_registry route scanner does not + // mistake this test call for a handler route arm. + let url = "/graph/edge?source=only-one"; + let (status, _) = app.call(Method::Delete, url, None); + assert_eq!(status, 400, "missing target param must 400"); + } + + // ── Namespace management API (#1181) ─────────────────────────────── + + struct NamespaceApp { + uteke: Mutex, + } + + impl NamespaceApp { + fn new() -> Self { + Self { + uteke: Mutex::new( + Uteke::open_with_backend(":memory:", None) + .expect("open in-memory uteke without embedder"), + ), + } + } + + fn call( + &self, + method: Method, + url: &str, + body: Option, + ) -> (u16, serde_json::Value) { + let mut req = match body { + Some(b) => { + let leaked: &'static str = Box::leak(b.into_boxed_str()); + TestRequest::new() + .with_method(method) + .with_path(url) + .with_body(leaked) + .into() + } + None => TestRequest::new().with_method(method).with_path(url).into(), + }; + let ctx = ReqCtx { + auth_token_hash: None, + read_only_token_hash: None, + cors_origins: Vec::new(), + recall_config: None, + extraction_config: None, + }; + let resp = route(&self.uteke, &ctx, &mut req); + let status = resp.status_code().0; + let bytes = resp.into_reader().into_inner(); + let json = serde_json::from_slice(&bytes).unwrap_or(serde_json::Value::Null); + (status, json) + } + + fn remember_in(&self, content: &str, namespace: &str) -> String { + let body = + serde_json::json!({ "content": content, "namespace": namespace }).to_string(); + let (status, resp) = self.call(Method::Post, "/remember", Some(body)); + assert_eq!(status, 200, "remember must succeed: {resp}"); + resp["id"].as_str().expect("id").to_string() + } + } + + #[test] + fn namespace_put_memory_moves_namespace() { + let app = NamespaceApp::new(); + let id = app.remember_in("to be moved", "alpha"); + + let body = serde_json::json!({ "id": id, "namespace": "beta" }).to_string(); + let (status, resp) = app.call(Method::Put, "/memory", Some(body)); + assert_eq!(status, 200, "PUT with namespace must succeed: {resp}"); + + // The move is visible via GET /memory. + let get_url = format!("/memory?id={id}"); + let (status, resp) = app.call(Method::Get, &get_url, None); + assert_eq!(status, 200); + assert_eq!(resp["namespace"], serde_json::json!("beta")); + + // The old name vanishes from listings (derived view). + let (_, list) = app.call(Method::Get, "/namespaces", None); + let names: Vec<&str> = list + .as_array() + .expect("namespaces array") + .iter() + .filter_map(|v| v.as_str()) + .collect(); + assert!(!names.contains(&"alpha"), "old namespace gone: {list:?}"); + assert!(names.contains(&"beta"), "{list:?}"); + } + + #[test] + fn namespace_rename_and_merge_endpoint() { + let app = NamespaceApp::new(); + app.remember_in("one", "old"); + app.remember_in("two", "old"); + app.remember_in("three", "existing"); + + let body = serde_json::json!({ "from": "old", "to": "existing" }).to_string(); + let (status, resp) = app.call(Method::Post, "/namespaces/rename", Some(body)); + assert_eq!(status, 200, "{resp}"); + assert_eq!(resp["moved"], serde_json::json!(2)); + assert_eq!(resp["target_existed"], serde_json::json!(true)); + assert_eq!(resp["from"], serde_json::json!("old")); + assert_eq!(resp["to"], serde_json::json!("existing")); + } + + #[test] + fn namespace_delete_refuse_returns_409() { + let app = NamespaceApp::new(); + app.remember_in("keep me", "busy"); + + let body = serde_json::json!({ "name": "busy" }).to_string(); + let (status, resp) = app.call(Method::Post, "/namespaces/delete", Some(body)); + assert_eq!( + status, 409, + "refuse on non-empty namespace must 409: {resp}" + ); + assert!( + resp["error"] + .as_str() + .unwrap_or_default() + .to_lowercase() + .contains("refus") + ); + } + + #[test] + fn namespace_delete_merge_removes_namespace() { + let app = NamespaceApp::new(); + app.remember_in("m one", "doomed"); + app.remember_in("m two", "doomed"); + + let body = serde_json::json!({ "name": "doomed", "strategy": "merge", "target": "safe" }) + .to_string(); + let (status, resp) = app.call(Method::Post, "/namespaces/delete", Some(body)); + assert_eq!(status, 200, "{resp}"); + assert_eq!(resp["affected"], serde_json::json!(2)); + assert_eq!(resp["empty"], serde_json::json!(true)); + + let (_, list) = app.call(Method::Get, "/namespaces", None); + let names: Vec<&str> = list + .as_array() + .expect("namespaces array") + .iter() + .filter_map(|v| v.as_str()) + .collect(); + assert!(!names.contains(&"doomed"), "{list:?}"); + assert!(names.contains(&"safe"), "{list:?}"); + } + + #[test] + fn namespace_with_counts_reports_lifecycle_split() { + let app = NamespaceApp::new(); + let id = app.remember_in("ghost food", "ghosted"); + let _ = id; + let body = serde_json::json!({ "name": "ghosted", "strategy": "deprecate" }).to_string(); + let (status, resp) = app.call(Method::Post, "/namespaces/delete", Some(body)); + assert_eq!(status, 200, "{resp}"); + assert_eq!(resp["affected"], serde_json::json!(1)); + assert_eq!(resp["empty"], serde_json::json!(false)); + + // URL built separately so the api_registry route scanner does not + // mistake this test call for a handler route arm. + let counts_url = "/namespaces?with_counts=true"; + let (_, counts) = app.call(Method::Get, counts_url, None); + let arr = counts.as_array().expect("counts array"); + let ghost = arr + .iter() + .find(|n| n["name"] == serde_json::json!("ghosted")) + .expect("ghost namespace stays listed"); + assert_eq!(ghost["active"], serde_json::json!(0)); + assert_eq!(ghost["deprecated"], serde_json::json!(1)); + assert_eq!(ghost["count"], serde_json::json!(1)); + } + + // ── Provenance API (#1172) ───────────────────────────────────────── + + #[test] + fn provenance_endpoint_reports_chain_and_hash() { + let app = NamespaceApp::new(); + let id = app.remember_in("provenance surface probe", "audit"); + + let get_url = format!("/provenance?id={id}"); + let (status, report) = app.call(Method::Get, &get_url, None); + assert_eq!(status, 200, "{report}"); + assert_eq!(report["id"], serde_json::json!(id)); + assert_eq!(report["namespace"], serde_json::json!("audit")); + assert_eq!(report["author_type"], serde_json::json!("agent")); + assert_eq!(report["trust_tier"], serde_json::json!("agent")); + assert_eq!(report["source_hash"], report["content_hash_now"]); + let events = report["events"].as_array().expect("events array"); + assert!( + events + .iter() + .any(|e| e["event_type"] == serde_json::json!("created")), + "created event must be in the chain: {report}" + ); + + // Unknown ID → 404. + let (status, _) = app.call( + Method::Get, + "/provenance?id=00000000-0000-4000-8000-000000000000", + None, + ); + assert_eq!(status, 404); + + // Missing param → 400. + let (status, _) = app.call(Method::Get, "/provenance", None); + assert_eq!(status, 400); + } +} + +// ── Contradiction ledger API (#1172 Fase 2) ───────────────────────── + +#[cfg(test)] +mod contradiction_api_tests { + use super::*; + use tiny_http::TestRequest; + + struct ContradictionApp { + uteke: Mutex, + } + + impl ContradictionApp { + fn new() -> Self { + // No embedder: storage-only endpoints, CI-safe without ONNX. + Self { + uteke: Mutex::new( + Uteke::open_with_backend(":memory:", None) + .expect("open in-memory uteke without embedder"), + ), + } + } + + fn call( + &self, + method: Method, + url: &str, + body: Option, + ) -> (u16, serde_json::Value) { + let mut req = match body { + Some(b) => { + let leaked: &'static str = Box::leak(b.into_boxed_str()); + TestRequest::new() + .with_method(method) + .with_path(url) + .with_body(leaked) + .into() + } + None => TestRequest::new().with_method(method).with_path(url).into(), + }; + let ctx = ReqCtx { + auth_token_hash: None, + read_only_token_hash: None, + cors_origins: Vec::new(), + recall_config: None, + extraction_config: None, + }; + let resp = route(&self.uteke, &ctx, &mut req); + let status = resp.status_code().0; + let bytes = resp.into_reader().into_inner(); + let json = serde_json::from_slice(&bytes).unwrap_or(serde_json::Value::Null); + (status, json) + } + + fn remember_in(&self, content: &str, namespace: &str) -> String { + let body = + serde_json::json!({ "content": content, "namespace": namespace }).to_string(); + let (status, resp) = self.call(Method::Post, "/remember", Some(body)); + assert_eq!(status, 200, "remember must succeed: {resp}"); + resp["id"] + .as_str() + .unwrap_or_else(|| panic!("remember response must carry id: {resp}")) + .to_string() + } + + fn supersede(&self, old: &str, new: &str) { + // Setup via the core API — the server has no supersede route; + // the surfaces under test are the ledger + undo endpoints. + self.uteke + .lock() + .expect("poisoned mutex") + .supersede(old, new, None) + .expect("supersede must succeed"); + } + } + + #[test] + fn ledger_lists_superseded_and_undo_restores() { + let app = ContradictionApp::new(); + let old = app.remember_in("deploy target is VM-1", "ops"); + let new = app.remember_in("deploy target is VM-2", "ops"); + app.supersede(&old, &new); + + // URL built separately so the api_registry route scanner does not + // mistake this test call for a handler route arm. + let list_url = "/contradictions?namespace=ops"; + let (status, ledger) = app.call(Method::Get, list_url, None); + assert_eq!(status, 200, "{ledger}"); + let entries = ledger.as_array().expect("ledger array"); + assert_eq!( + entries.len(), + 1, + "one supersession must be listed: {ledger}" + ); + assert_eq!(entries[0]["id"], serde_json::json!(old)); + let reason = entries[0]["deprecate_reason"].as_str().unwrap_or(""); + assert!( + reason.contains("superseded"), + "reason must mention superseded: {reason}" + ); + + // Undo via POST body. + let undo_body = serde_json::json!({ "id": old }).to_string(); + let (status, resp) = app.call(Method::Post, "/contradictions/undo", Some(undo_body)); + assert_eq!(status, 200, "{resp}"); + assert_eq!(resp["undone"], serde_json::json!(true)); + assert_eq!(resp["restored"], serde_json::json!(old)); + assert_eq!(resp["was_superseded_by"], serde_json::json!(new)); + + // Ledger is clean after the undo. + let (status, ledger) = app.call(Method::Get, list_url, None); + assert_eq!(status, 200); + assert_eq!( + ledger.as_array().expect("ledger array").len(), + 0, + "undo must clear the ledger: {ledger}" + ); + } + + #[test] + fn undo_unknown_supersession_is_404() { + let app = ContradictionApp::new(); + let id = app.remember_in("never superseded", "ops"); + + let body = serde_json::json!({ "id": id }).to_string(); + let (status, resp) = app.call(Method::Post, "/contradictions/undo", Some(body)); + assert_eq!(status, 404, "undo without a supersession must 404: {resp}"); + + // Malformed body → 400. + let (status, _) = app.call( + Method::Post, + "/contradictions/undo", + Some("not-json".into()), + ); + assert_eq!(status, 400); + } +} + +// ── Explain recall API (#1160) ────────────────────────────────────── + +#[cfg(test)] +mod explain_recall_api_tests { + use super::*; + use tiny_http::TestRequest; + + struct ExplainApp { + uteke: Mutex, + } + + impl ExplainApp { + fn new() -> Self { + // No embedder: tests use the fts5 strategy, which needs no + // query embedding (CI-safe without ONNX). + Self { + uteke: Mutex::new( + Uteke::open_with_backend(":memory:", None) + .expect("open in-memory uteke without embedder"), + ), + } + } + + fn call( + &self, + method: Method, + url: &str, + body: Option, + ) -> (u16, serde_json::Value) { + let mut req = match body { + Some(b) => { + let leaked: &'static str = Box::leak(b.into_boxed_str()); + TestRequest::new() + .with_method(method) + .with_path(url) + .with_body(leaked) + .into() + } + None => TestRequest::new().with_method(method).with_path(url).into(), + }; + let ctx = ReqCtx { + auth_token_hash: None, + read_only_token_hash: None, + cors_origins: Vec::new(), + recall_config: None, + extraction_config: None, + }; + let resp = route(&self.uteke, &ctx, &mut req); + let status = resp.status_code().0; + let bytes = resp.into_reader().into_inner(); + let json = serde_json::from_slice(&bytes).unwrap_or(serde_json::Value::Null); + (status, json) + } + + fn remember_in(&self, content: &str, namespace: &str) -> String { + let body = + serde_json::json!({ "content": content, "namespace": namespace }).to_string(); + let (status, resp) = self.call(Method::Post, "/remember", Some(body)); + assert_eq!(status, 200, "remember must succeed: {resp}"); + resp["id"] + .as_str() + .unwrap_or_else(|| panic!("remember response must carry id: {resp}")) + .to_string() + } + } + + #[test] + fn explain_fts5_returns_signals_and_guards() { + let app = ExplainApp::new(); + let fox_id = app.remember_in("The quick brown fox jumps over the lazy dog", "explain-ns"); + app.remember_in( + "Completely unrelated content about gardening tools", + "explain-ns", + ); + + // Body built separately so the api_registry route scanner does not + // mistake this literal for a handler route arm. + let body = serde_json::json!({ + "query": "quick brown fox", + "limit": 5, + "namespace": "explain-ns", + "strategy": "fts5", + "explain": true + }) + .to_string(); + let (status, resp) = app.call(Method::Post, "/recall", Some(body)); + assert_eq!(status, 200, "{resp}"); + let arr = resp.as_array().expect("explained results array"); + assert!(!arr.is_empty(), "fts5 must find the fox"); + let first = &arr[0]; + assert_eq!(first["memory"]["id"], serde_json::json!(fox_id)); + let ex = &first["explanation"]; + assert_eq!(ex["strategy"], serde_json::json!("fts5")); + assert_eq!(ex["fts_rank"], serde_json::json!(1)); + assert!(ex["final_score"].is_number(), "final_score must exist"); + + // explain + search_type → 400 (memory-only feature). + let bad = serde_json::json!({ + "query": "quick brown fox", + "search_type": "all", + "explain": true + }) + .to_string(); + let (status, _) = app.call(Method::Post, "/recall", Some(bad)); + assert_eq!(status, 400, "explain + search_type must 400"); + + // explain + at → 400. + let bad = serde_json::json!({ + "query": "quick brown fox", + "at": "2026-01-01T00:00:00Z", + "explain": true + }) + .to_string(); + let (status, _) = app.call(Method::Post, "/recall", Some(bad)); + assert_eq!(status, 400, "explain + at must 400"); + + // explain + entity → 400 (filter would be silently dropped). + let bad = serde_json::json!({ + "query": "quick brown fox", + "entity": "staging", + "explain": true + }) + .to_string(); + let (status, _) = app.call(Method::Post, "/recall", Some(bad)); + assert_eq!(status, 400, "explain + entity must 400"); + } +} + +// ── /list pagination metadata (#1188) ─────────────────────────────── + +#[cfg(test)] +mod list_pagination_tests { + use super::*; + use tiny_http::TestRequest; + + struct ListApp { + uteke: Mutex, + } + + impl ListApp { + fn new() -> Self { + Self { + uteke: Mutex::new( + Uteke::open_with_backend(":memory:", None) + .expect("open in-memory uteke without embedder"), + ), + } + } + + fn call( + &self, + method: Method, + url: &str, + body: Option, + ) -> (u16, serde_json::Value) { + let mut req = match body { + Some(b) => { + let leaked: &'static str = Box::leak(b.into_boxed_str()); + TestRequest::new() + .with_method(method) + .with_path(url) + .with_body(leaked) + .into() + } + None => TestRequest::new().with_method(method).with_path(url).into(), + }; + let ctx = ReqCtx { + auth_token_hash: None, + read_only_token_hash: None, + cors_origins: Vec::new(), + recall_config: None, + extraction_config: None, + }; + let resp = route(&self.uteke, &ctx, &mut req); + let status = resp.status_code().0; + let bytes = resp.into_reader().into_inner(); + let json = serde_json::from_slice(&bytes).unwrap_or(serde_json::Value::Null); + (status, json) + } + + fn remember_in(&self, content: &str, namespace: &str) { + let body = + serde_json::json!({ "content": content, "namespace": namespace }).to_string(); + let (status, resp) = self.call(Method::Post, "/remember", Some(body)); + assert_eq!(status, 200, "seed remember must succeed: {resp}"); + } + } + + #[test] + fn list_meta_envelope_and_bare_default() { + let app = ListApp::new(); + for i in 0..7 { + app.remember_in(&format!("pagination probe {i}"), "pg"); + } + + // Default (no include_meta): bare array — unchanged shape. + let body = serde_json::json!({ "namespace": "pg", "limit": 100 }).to_string(); + let (status, resp) = app.call(Method::Post, "/list", Some(body)); + assert_eq!(status, 200); + assert!(resp.is_array(), "default must stay a bare array: {resp}"); + assert_eq!(resp.as_array().unwrap().len(), 7); + + // include_meta: envelope with total/has_more/next_offset. + // URL built separately so the api_registry route scanner does not + // mistake this literal for a handler route arm. + let body = serde_json::json!({ + "namespace": "pg", + "limit": 3, + "offset": 0, + "include_meta": true + }) + .to_string(); + let (status, resp) = app.call(Method::Post, "/list", Some(body)); + assert_eq!(status, 200); + assert!(resp.is_object(), "envelope expected: {resp}"); + assert_eq!(resp["total"], serde_json::json!(7)); + assert_eq!(resp["has_more"], serde_json::json!(true)); + assert_eq!(resp["next_offset"], serde_json::json!(3)); + assert_eq!(resp["memories"].as_array().expect("page rows").len(), 3); + + // Last page: has_more false, next_offset null. + let body = serde_json::json!({ + "namespace": "pg", + "limit": 10, + "offset": 6, + "include_meta": true + }) + .to_string(); + let (status, resp) = app.call(Method::Post, "/list", Some(body)); + assert_eq!(status, 200); + assert_eq!(resp["has_more"], serde_json::json!(false)); + assert!(resp["next_offset"].is_null()); + assert_eq!( + resp["memories"].as_array().expect("page rows").len(), + 1, + "offset 6 of 7 rows returns the last row" + ); + + // include_meta + at: `at` wins — bare array (documented). + let body = serde_json::json!({ + "namespace": "pg", + "include_meta": true, + "at": "2100-01-01T00:00:00Z" + }) + .to_string(); + let (status, resp) = app.call(Method::Post, "/list", Some(body)); + assert_eq!(status, 200); + assert!(resp.is_array(), "at-mode stays a bare array: {resp}"); + } } diff --git a/crates/uteke-server/src/types.rs b/crates/uteke-server/src/types.rs index 45032552..b1a21087 100644 --- a/crates/uteke-server/src/types.rs +++ b/crates/uteke-server/src/types.rs @@ -252,6 +252,11 @@ pub struct RecallRequest { /// Invalid values return HTTP 400. #[serde(default)] pub strategy: Option, + /// Explain mode (#1160): return per-result ranking signals alongside + /// each memory. Memory-only recall — rejected (400) together with + /// search_type/unified, at, before/after. + #[serde(default)] + pub explain: bool, /// Temporal range filter: only return memories created at or after this /// RFC3339 timestamp (#902). #[serde(default)] @@ -288,6 +293,13 @@ pub struct ListParams { /// Time-travel: list memories that existed at this RFC3339 timestamp. #[serde(default)] pub at: Option, + /// Pagination metadata (#1188): when true, respond with an envelope + /// `{memories, total, has_more, next_offset}` instead of the bare array. + /// Default false — the bare-array shape is unchanged for existing + /// clients. Not supported with `at` (point-in-time listing returns the + /// bare array regardless). + #[serde(default)] + pub include_meta: bool, } #[cfg_attr(feature = "docgen", derive(schemars::JsonSchema))] @@ -500,6 +512,39 @@ pub struct MemoryUpdateRequest { /// Set memory type (fact, procedure, preference, decision, context, note, insight, reference, event). #[serde(default)] pub memory_type: Option, + /// Move the memory to this namespace (#1181). Plain column update — no re-embed. + #[serde(default)] + pub namespace: Option, +} + +// ── Namespace Management Types (#1181) ───────────────────────────────────── + +#[cfg_attr(feature = "docgen", derive(schemars::JsonSchema))] +#[derive(Deserialize)] +pub struct NamespaceRenameRequest { + /// Current namespace name. + pub from: String, + /// New namespace name. If it already exists, this is a merge. + pub to: String, +} + +#[cfg_attr(feature = "docgen", derive(schemars::JsonSchema))] +#[derive(Deserialize)] +pub struct NamespaceDeleteRequest { + /// Namespace to delete. + pub name: String, + /// What happens to its memories: `refuse` (default — 409-style error while + /// any memory references the name), `merge` (move all memories to `target`), + /// or `deprecate` (soft-delete all memories — restorable, never hard-deleted). + #[serde(default = "default_namespace_delete_strategy")] + pub strategy: String, + /// Target namespace when strategy is `merge`. + #[serde(default)] + pub target: Option, +} + +fn default_namespace_delete_strategy() -> String { + "refuse".to_string() } // ── Pin Types ───────────────────────────────────────────────────────────── diff --git a/docker-compose.yml b/docker-compose.yml index f8c18c0d..2d0f2aa9 100644 --- a/docker-compose.yml +++ b/docker-compose.yml @@ -12,6 +12,10 @@ services: environment: # Auth (optional — set UTEKE_AUTH_TOKEN env var to enable) # - UTEKE_AUTH_TOKEN + # Vector engine (v0.17.0+): usearch (default) or vecq. + # Both engines ship in the image; switching rebuilds the index from + # SQLite on next start (data is never touched). + - UTEKE_VECTOR_BACKEND=${UTEKE_VECTOR_BACKEND:-usearch} restart: unless-stopped healthcheck: # curl is pre-installed in the uteke Docker image (Dockerfile runtime stage) diff --git a/docs/api-reference.md b/docs/api-reference.md index f97095e2..e40d2bbe 100644 --- a/docs/api-reference.md +++ b/docs/api-reference.md @@ -52,7 +52,25 @@ Returns the agent-facing memory tools guide for system prompt injection (#1010). #### 🟢 `GET` `/namespaces` -List all namespaces in the memory store +List all namespaces in the memory store. `?with_counts=true` adds `count` (total) plus `active`/`deprecated` breakdown fields (#1181). + +*Related: `1181`* + +#### 🟡 `POST` `/namespaces/rename` + +Rename a namespace (`{from, to}`). When `to` already exists this is a merge; the old name vanishes (derived view). Returns `{from, to, moved, target_existed}` (#1181). + +**Request body**: [`NamespaceRenameRequest`](#namespacerenamerequest) + +*Related: `1181`* + +#### 🟡 `POST` `/namespaces/delete` + +Delete a namespace with an explicit strategy for its memories (`{name, strategy, target?}`): `refuse` (default — 409 while any memory references the name), `merge` (move all memories to `target`), or `deprecate` (soft-delete — restorable, never hard-deleted) (#1181). + +**Request body**: [`NamespaceDeleteRequest`](#namespacedeleterequest) + +*Related: `1181`* #### 🟢 `GET` `/stats` @@ -82,6 +100,26 @@ Update an existing memory's content and/or metadata. Get graph edges for a memory. Accepts `?id=...` query param. +#### 🟢 `GET` `/provenance` + +Full provenance report for a memory (#1172): author/source fields, trust tier, source hash at write vs live-recomputed content hash (tamper evidence), and the full timeline event chain with actor + evidence. Accepts `?id=...` query param. + +*Related: `1172`* + +#### 🟢 `GET` `/contradictions` + +List superseded-but-not-restored memories (#1172) — the auditable contradiction resolution ledger (deprecated rows carrying a live superseded_by edge). Accepts `?namespace=...&limit=...`. + +*Related: `1172`* + +#### 🟡 `POST` `/contradictions/undo` + +Undo a contradiction resolution (#1172): restore the retired memory to active, remove the supersession pair, and record supersession_undone provenance events. Body: `{id}`. + +**Request body**: [`ContradictionUndoRequest`](#contradictionundorequest) + +*Related: `1172`* + #### 🟡 `POST` `/lifecycle/cycle` Run lifecycle aging cycle: deprecate old memories, optionally prune expired ones. @@ -233,13 +271,17 @@ Get memories that reference a specific document. #### 🟡 `POST` `/graph/edge` -Add a directed edge between two memories. +Add a directed edge between two memories. Accepts memory IDs (a linked graph node is ensured automatically, #1180) or existing graph node IDs. Returns `{ok, source_node, target_node}`. **Request body**: [`GraphEdgeRequest`](#graphedgerequest) +*Related: `1180`* + #### 🔴 `DELETE` `/graph/edge` -Remove an edge between two memories. Accepts `?from=...&to=...` query params. +Remove an edge between two nodes. Accepts memory IDs or graph node IDs via `?source=...&target=...` query params (#1180). + +*Related: `1180`* #### 🟢 `GET` `/edges` @@ -262,7 +304,7 @@ Store a new memory. Accepts content, tags, namespace, type, metadata. #### 🟡 `POST` `/recall` -Semantic search — recall memories by meaning. Returns ranked results. +Semantic search — recall memories by meaning. Returns ranked results. Set `explain: true` (#1160) to include per-result ranking signals (vector similarity/rank, RRF contributions, boosts); memory-only recall. *Excludes deprecated memories from results.* @@ -278,7 +320,7 @@ Keyword search — find memories by matching words in content/tags. #### 🟡 `POST` `/list` -List memories with optional filters (namespace, tags, sort, limit, offset). +List memories with optional filters (namespace, tags, sort, limit, offset). Set `include_meta: true` (#1188) for a pagination envelope `{memories, total, has_more, next_offset}` instead of the bare array (not supported with `at`). *Excludes deprecated memories from results.* @@ -367,7 +409,7 @@ List all memories in a room (chronological). Accepts `?room_id=...` query param. #### 🔴 `DELETE` `/room/delete` -Delete a room and all its memories. Accepts `?room_id=...` query param. +Delete a room (unlink-only): room links are removed, memories and documents are preserved. Response: `{ deleted, unlinked_memories }`. Accepts `?room_id=...` query param. #### 🟡 `POST` `/room/document` @@ -566,6 +608,11 @@ features on the actual server capability rather than a local CLI probe. | | Field | Type | Required | Description | |-------|------|----------|-------------| | `at` | any | No | Time-travel: list memories that existed at this RFC3339 timestamp. | +| `include_meta` | `boolean` | No | Pagination metadata (#1188): when true, respond with an envelope +`{memories, total, has_more, next_offset}` instead of the bare array. +Default false — the bare-array shape is unchanged for existing +clients. Not supported with `at` (point-in-time listing returns the +bare array regardless). | | `limit` | `integer` | No | | | `namespace` | any | No | | | `offset` | `integer` | No | | @@ -607,6 +654,7 @@ Request for memory feedback / trust scoring (#718). | `importance` | any | No | Set importance score (0.0–1.0). | | `memory_type` | any | No | Set memory type (fact, procedure, preference, decision, context, note, insight, reference, event). | | `metadata` | any | No | Replace metadata entirely with this object. | +| `namespace` | any | No | Move the memory to this namespace (#1181). Plain column update — no re-embed. | | `pinned` | any | No | Set pinned state. | | `tags` | any | No | Replace tags entirely with this list. | @@ -650,6 +698,9 @@ RFC3339 timestamp (#902). | When true, populates `linked_doc_slugs` on memory results and `linked_memory_ids` on document results. | | `entity` | any | No | Filter by entity metadata. | +| `explain` | `boolean` | No | Explain mode (#1160): return per-result ranking signals alongside +each memory. Memory-only recall — rejected (400) together with +search_type/unified, at, before/after. | | `limit` | `integer` | No | | | `min_score` | any | No | Minimum similarity score (0.0-1.0). Results below are filtered. Default: 0.0 (no filtering). Use `strict=true` for 0.5 default (#995). | diff --git a/docs/benchmarks.md b/docs/benchmarks.md index 7a7bf55e..942b71dc 100644 --- a/docs/benchmarks.md +++ b/docs/benchmarks.md @@ -116,6 +116,10 @@ the same data. - The FTS5-only bar is an ablation of our own system, not a competitor. - Aggregate metrics + per-type breakdown: `benchmarks/longmemeval/RESULTS.md` in this repo. Raw per-question results are kept on the benchmark Modal volume (`uteke-longmemeval`, `default/` prefix), not committed here. +### Reproducibility + +An independent local re-run (2026-09-01) of 108 of the 500 questions on a 4-core ARM desktop reproduced the published Modal x86 run: **107/108 questions produced identical per-question rankings**. The single difference was an adjacent-rank near-tie (identical top-10 set, one gold session swapped ranks 5-6) from cross-architecture floating-point noise. Aggregate R@5 on the subset: 96.7% / 99.4% (re-run) vs 96.7% / 100.0% (published). See the [Independent Reproduction section in RESULTS.md](../benchmarks/longmemeval/RESULTS.md) for the full table and reproduction command. + ## Environment | Component | Details | diff --git a/docs/cli-reference.md b/docs/cli-reference.md index 6bb66a3b..5da24a3d 100644 --- a/docs/cli-reference.md +++ b/docs/cli-reference.md @@ -120,6 +120,8 @@ uteke recall "config" --category infrastructure --limit 5 uteke recall "API architecture" --type doc # Search memories only (backward compatible) uteke recall "deployment" --type memory +# Explain mode (#1160): show the ranking signals behind each result +uteke recall "database caching" --explain ``` | Flag | Description | @@ -130,6 +132,7 @@ uteke recall "deployment" --type memory | `--category ` | Filter results to a specific category | | `--content-format ` | Content display: `auto` (detect), `text`, `json` (pretty-print JSON memories) | | `--where ` | Filter by JSON field on structured memories (e.g. `--where role=CTO`) | +| `--explain` | Show the ranking signals behind each result (#1160): final score, strategy, vector similarity + rank, FTS rank, RRF score with per-channel fusion contributions, and jaccard/salience/recency/graph boost deltas. Memory-only — not available with `--type doc` | | `--json` | Output as JSON array | ## uteke search @@ -262,9 +265,57 @@ uteke lifecycle restore See [Configuration → Memory Lifecycle](/configuration#memory-lifecycle) for lifecycle config options. +## uteke provenance + +Show the full provenance report for a memory (#1172) — who wrote it, from +what source, its trust tier, whether the content still matches the hash +recorded at write time, and the complete event chain with actor + +evidence. + +```bash +# Human-readable audit report +uteke provenance + +# JSON (for tooling) +uteke provenance --json +``` + +Hash verdicts: `✓ matches` (content unchanged since write), `✗ MISMATCH` +(content modified after write — investigate), `—` (memory predates +schema v18; hash will be recorded on next content update). + +## uteke supersede + +Resolve a conflict: mark the old memory superseded by a newer one (#1053) — +wires the supersession edge pair, soft-deprecates the old memory, and records +the resolution in the contradiction ledger (#1172). + +```bash +uteke supersede --reason "decision pivot" +``` + +## uteke contradictions + +Inspect the contradiction resolution ledger (#1172) — memories that were +superseded by conflict resolution and are not restored. + +```bash +# List the resolution ledger +uteke contradictions list + +# Filter by namespace, cap entries +uteke contradictions list --namespace ops --limit 20 + +# JSON (for tooling) +uteke contradictions list --json + +# Restore a superseded memory (full UUID or unambiguous prefix) +uteke contradictions undo +``` + ## uteke namespace -Manage namespaces — list, inspect, and switch defaults. +Manage namespaces — list, inspect, switch defaults, and manage members (#1181). ```bash # List all namespaces with counts @@ -275,8 +326,23 @@ uteke namespace stats my-agent # Switch default namespace (saved to config) uteke namespace switch my-agent + +# Move a memory to another namespace (#1181 — no re-embed) +uteke namespace move other-ns + +# Rename a namespace — merges into an existing target (#1181) +uteke namespace rename old-ns new-ns + +# Delete a namespace with an explicit strategy for its memories (#1181) +uteke namespace delete temp-ns --strategy merge --target archive --confirm +uteke namespace delete ghost-ns --strategy deprecate --confirm ``` +Delete strategies: `refuse` (default — refuses while any memory references the +name), `merge` (move all memories to `--target`, the name vanishes), or +`deprecate` (soft-delete — restorable via `uteke lifecycle`/promote, never +hard-deleted). `--confirm` is required for `delete`. + Namespace resolution order: `--namespace flag` → `UTEKE_NAMESPACE` env → `uteke.toml` → `"default"` ## uteke tags diff --git a/docs/configuration.md b/docs/configuration.md index f8ee60ae..a3b0c2a7 100644 --- a/docs/configuration.md +++ b/docs/configuration.md @@ -44,6 +44,14 @@ host = "127.0.0.1" # Server port port = 8767 + +[vector] +# Vector search engine when the binary ships BOTH engines (official builds): +# "usearch" (default — HNSW, C++ FFI, fastest) or "vecq" (training-free +# quantization, zero C++ dep — mobile/slim builds). Switching engines on an +# existing store auto-rebuilds the index from SQLite on next open (#1168). +# Env override: UTEKE_VECTOR_BACKEND=usearch|vecq +backend = "usearch" ``` ## Server Mode @@ -267,6 +275,7 @@ Resolution order (highest priority first): | `UTEKE_NAMESPACE` | `[store] namespace` | `default` | Default namespace (applied in CLI) | | `UTEKE_AUTH_TOKEN` | — | — | Server auth token (applied in server) | | `UTEKE_LOG_LEVEL` | `[logging] level` | `warn` | Log level (trace/debug/info/warn/error) | +| `UTEKE_VECTOR_BACKEND` | `[vector] backend` | compiled-in default | Vector engine when both are compiled in: `usearch`, `vecq`. Ignored (with a warning) on single-engine builds or unknown values. Switching engines on an existing store auto-rebuilds the index from SQLite | | `UTEKE_SERVER_HOST` | `[server] host` | `127.0.0.1` | Server bind address | | `UTEKE_SERVER_PORT` | `[server] port` | `8767` | Server port | | `UTEKE_RECALL_MIN_SCORE` | `[recall] min_score` | `0.3` | Default similarity threshold | diff --git a/docs/docker.md b/docs/docker.md index e1c924d8..22c5160f 100644 --- a/docs/docker.md +++ b/docs/docker.md @@ -101,10 +101,25 @@ docker run -v /path/to/uteke:/data ... The volume contains: - `uteke.db` — SQLite database (memories, metadata, FTS5) -- `uteke_index.usearch` — HNSW vector index +- `uteke_index.usearch` — HNSW vector index (default engine) +- `uteke_index.vecq` — quantized index (created if you switch engines) - `uteke_index.keys` — Index key mapping - `models/embeddinggemma-q4/` — ONNX embedding model (~188MB) +### Choosing the vector engine (v0.17.0+) + +Both engines ship in the image. Pick one via env var — switching leaves your +data untouched; the new engine rebuilds its index from SQLite on the next start: + +```yaml +environment: + - UTEKE_VECTOR_BACKEND=vecq # or "usearch" (default) +``` + +`usearch` (HNSW) has the best query latency at scale; `vecq` (4-bit + residual, +pure Rust) builds indexes ~16x faster with ~3x smaller files. See +[configuration](configuration.md#environment-variables) for details. + ## Multi-Architecture Images are built for: