|
1 | 1 | # Corbits Memory — Architecture |
2 | 2 |
|
3 | | -A memory add / find / ask / recent SDK that mounts onto an Interchange hub. The |
4 | | -host owns auth, tenancy, and the process; this library owns the memory / |
5 | | -vector plane and the routes that read and write it. |
| 3 | +A memory **add / search / list** SDK that mounts onto an Interchange hub. The |
| 4 | +host owns auth, tenancy, and the process; this library owns the durable memory |
| 5 | +plane and the protected routes that read and write it. |
6 | 6 |
|
7 | 7 | ## Why an SDK, not a service |
8 | 8 |
|
9 | | -The memory store was originally built inside a larger backend. It turned out |
10 | | -to be cleanly detachable, and then cleanly *mountable*: |
| 9 | +The store was detachable from a larger backend, then mountable: |
11 | 10 |
|
12 | | -- No memory table has a foreign key into any control-plane table — every |
13 | | - cross-reference (`tenant_id`, `principal_id`, source refs) is plain `text`. |
14 | | -- Embedding and reranking go out as plain HTTP to configured model endpoints, |
15 | | - not through any agent runtime. |
16 | | -- The ACL rule is a self-contained scope stored on the row, not a join against |
17 | | - a grant engine. |
| 11 | +- No memory table has a foreign key into any control-plane table — cross-refs |
| 12 | + (`tenant_id`, `principal_id`, source refs) are plain `text`. |
| 13 | +- Embedding and reranking go out as plain HTTP to configured model endpoints. |
| 14 | +- Document access is Interchange grant tags on the row (`accessTags` + creator), |
| 15 | + not a private ACL engine inside this package. |
18 | 16 |
|
19 | | -So the library needs nothing but a pgvector Postgres and an embed/rerank |
20 | | -endpoint. It ships as `createMemory(opts)` — pass `app` to register HTTP. The host passes its |
| 17 | +It ships as `createMemory({ app, … })`: the host passes its Hono app and grant |
| 18 | +store; the library registers routes, reads identity from request context, and |
| 19 | +talks to its DocumentStore. No second server. |
21 | 20 |
|
22 | | -Hono app and its grant store; the library mounts its routes, reads identity from |
23 | | -the request context, and talks to its own vector store. No second server, no |
24 | | -HTTP hop. |
| 21 | +## Product path |
| 22 | + |
| 23 | +``` |
| 24 | +tools / ingestion → /api/tenants/:tenantId/memory/* → Memory plane → DocumentStore |
| 25 | + ↑ |
| 26 | + Interchange auth + principal + grants |
| 27 | +``` |
| 28 | + |
| 29 | +Mount is intentionally small. The host already has `app`, grants, and |
| 30 | +principal middleware; memory only needs to be handed those and the vector |
| 31 | +config (or an injected store). |
25 | 32 |
|
26 | 33 | ## Boundaries |
27 | 34 |
|
28 | | -- **Runtime**: Bun + Hono, mounted on the host's app. **DB**: its own pgvector |
29 | | - Postgres (`KNOWLEDGE_DATABASE_URL`). **Types**: arktype at every route |
30 | | - boundary. |
31 | | -- **No auth of its own.** The SDK authenticates nothing. Interchange resolves |
32 | | - the caller (session, `cke_` API key, or MCP OAuth) and puts `principal` + |
33 | | - `tenant` on the request context; each mounted route reads identity from there |
34 | | - (`caller(c)` → `scopeId = principal.tenantId`, `subjectId = principal.id`). |
35 | | -- **Grants delegate to the host.** Pass `grants` (`{ grantStore, |
36 | | - conditionRegistry }`) and the SDK guards routes with Interchange's |
37 | | - `createRequireGrant`. |
38 | | -- **Dependencies** are public npm only — `@intx/hub-api` (`TenantEnv`, |
39 | | - `createRequireGrant`), `@intx/authz` (`authorize`), `@intx/log`, Hono, |
40 | | - Drizzle, arktype, `postgres`, `hono-openapi`. Eight total. LGPL-2.1-licensed — |
41 | | - see `LICENSE`. |
42 | | - |
43 | | -## Identity — read from context, stored as data |
44 | | - |
45 | | -1. **Who is calling** is the request principal, read off the Interchange |
46 | | - context. Clients never send `tenant_id`/`principal_id` — the handlers read |
47 | | - only content fields (title/text/query/limit/access_tags/share) and take identity from context. |
48 | | -2. **What is stored** is opaque data on every record: `tenant_id`, |
49 | | - `principal_id`, `created_by_kind` (human/agent/system), `source_class`, and |
50 | | - relations (the edge graph). Every query is scoped by `tenant_id` first; then |
51 | | - document access uses Interchange grant tags (`accessTags` + creator). |
52 | | - |
53 | | -Cross-tenant isolation is enforced at query time by `tenant_id`; document-level |
54 | | -access is grant tags via `@intx/authz` (creator always allowed). This is the |
55 | | -trust model. |
56 | | - |
57 | | -## Layers |
58 | | - |
59 | | -- `raw_capture` — immutable, append-only original content. The replay substrate. |
60 | | -- `derived` — chunks / embeddings / authority / edges, all derived from |
61 | | - `raw_capture`. |
62 | | -- `transform_config` + replay — a named, versioned transform (chunk strategy, |
63 | | - embed model, rerank endpoint, authority weights, MMR λ) that rebuilds the |
64 | | - derived layer from raw without re-fetching source. |
| 35 | +- **Runtime**: Bun + Hono, mounted on the host app. **DB**: own pgvector |
| 36 | + Postgres (`KNOWLEDGE_DATABASE_URL`) unless `documentStore` is injected. |
| 37 | + **Types**: arktype at every route boundary. |
| 38 | +- **No auth of its own.** Interchange resolves the caller and puts `principal` |
| 39 | + + `tenant` on context; routes read identity from there |
| 40 | + (`tenantId = principal.tenantId`, `principalId = principal.id`). |
| 41 | +- **Grants delegate to the host.** Pass `grantStore` + `conditionRegistry`; |
| 42 | + routes use `createRequireGrant("memory", action)`. |
| 43 | +- **Dependencies**: `@intx/hub-api`, `@intx/authz`, `@intx/log`, Hono, Drizzle, |
| 44 | + arktype, `postgres`, `hono-openapi`. LGPL-2.1 — see `LICENSE`. |
65 | 45 |
|
66 | | -## Mounted surface |
| 46 | +## Identity — context in, data out |
| 47 | + |
| 48 | +1. **Who is calling** is the request principal. Clients never send |
| 49 | + `tenant_id` / `principal_id` on the body. |
| 50 | +2. **What is stored** is opaque data: `tenant_id`, `principal_id`, |
| 51 | + `created_by_kind`, `access_tags`, source refs. Queries scope by `tenant_id` |
| 52 | + first; document access is grant tags + creator. |
67 | 53 |
|
68 | | -`createMemory({ app })` adds, under the host app: |
| 54 | +## Layers (default pgvector store) |
69 | 55 |
|
70 | | -- `POST /api/memory/add` — ingest a note (raw + derive). |
71 | | -- `POST /api/memory/search` — hybrid retrieval: FTS + dense (pgvector) → RRF |
72 | | - fusion → cross-encoder rerank → bounded authority/recency boosts → MMR; |
73 | | - optional live `SourceProvider` merge (fail-soft). |
74 | | -- `GET /api/memory/list` — recent documents for the caller's scope, |
75 | | - filtered with the same grant-tag access as local search (`canAccessDocument`). |
| 56 | +- `raw_capture` — immutable original content (replay substrate). |
| 57 | +- `derived` — chunks / embeddings / authority / edges from raw. |
| 58 | +- `transform_config` + replay — rebuild derived from raw without re-fetch. |
| 59 | + |
| 60 | +Injected DocumentStores own their own persistence model; the plane still |
| 61 | +exposes the same three verbs. |
| 62 | + |
| 63 | +## Mounted surface |
76 | 64 |
|
77 | | -It also returns an in-process `Memory` (`add`, `search`, `list`, `close`). |
78 | | -There is no product `ask` / `remember` / `recall` and no host-injected |
79 | | -`generate` on the plane — inference is host-owned and ephemeral (call your |
80 | | -model, then `add` / `search`). |
| 65 | +`createMemory({ app })` registers: |
81 | 66 |
|
82 | | -MCP is not part of this package — mount `@corbitsdev/hono-openapi-mcp` to expose |
83 | | -these routes as MCP tools. |
| 67 | +- `POST /api/tenants/:tenantId/memory/add` — ingest (raw + derive on the default store). |
| 68 | +- `POST /api/tenants/:tenantId/memory/search` — hybrid retrieval (FTS + dense → RRF → rerank → |
| 69 | + authority/recency → MMR); optional live `SourceProvider` merge (fail-soft). |
| 70 | +- `GET /api/tenants/:tenantId/memory/list` — recent documents, same grant-tag filter as local |
| 71 | + search. |
84 | 72 |
|
85 | | -External ingestion (Linear, GitHub, …) is not a route here — the host |
86 | | -authenticates the forwarder to Interchange and calls `plane.add` / a |
87 | | -`SourceProvider` mapper, or mounts HTTP add after its own auth. |
| 73 | +Returns an in-process `Memory` (`add`, `search`, `list`, `close`) for host |
| 74 | +workers and ingestion modules that already resolved identity. |
88 | 75 |
|
89 | | -Legacy paths `/capture`, `/search` (old knowledge), `/timeline`, `/find`, |
90 | | -`/ask`, `/recent` are not mounted (hard cutover). |
| 76 | +**Agent tools are not in this package.** Routes are OpenAPI-described |
| 77 | +(`describeRoute`). The host mounts `@corbitsdev/hono-openapi-mcp` (or any |
| 78 | +OpenAPI→tools bridge) so agents call these routes under Interchange auth. |
91 | 79 |
|
92 | 80 | ## Provenance |
93 | 81 |
|
94 | | -The framework-agnostic core (chunk strategies, embed client + model registry, |
95 | | -authority weighting, hybrid search, MMR, rerank client, ingestion adapters) was |
96 | | -extracted from an internal RAG implementation and generalized. The persistence |
97 | | -and the mountable surface are native to this repo. |
| 82 | +Framework-agnostic core (chunking, embed/rerank clients, hybrid search, MMR) |
| 83 | +was extracted from an internal RAG implementation. Persistence and the |
| 84 | +mountable surface are native to this repo. |
0 commit comments