wikimem

English | 简体中文 | Docs

File-first memory for AI agents: RecallFiles + wiki-links over plain markdown. No database, no embedding model, no docker — pip install wikimem and it works.

Install

pip install wikimem        # the default — everything works out of the box
pip install "wikimem[all]" # optional enhancements included, if you'd rather not choose

There are no modes. wikimem is one pipeline. Extras only unlock optional enhancements, they activate automatically, and they never conflict with each other — installing all of them changes nothing until you actually use them.

Install	Adds	Use case
wikimem	nothing — zero dependencies	Always fully works: storage, BM25 retrieval (Chinese via char-bigrams), wiki-links, journal
wikimem[zh]	jieba	Sharper Chinese keyword recall than bigrams — picked up automatically once installed, nothing to configure
wikimem[embed]	httpx + numpy	Semantic recall (match by meaning, not wording) — only active when you pass an embedder; endpoint down → BM25 carries on
wikimem[all]	both of the above	The "don't make me think" option

Design rules

1. Markdown files are the only source of truth. One RecallFile per topic under wiki/ (memory/wiki/preferences.md), one ## heading per item; diary events live as per-day files under diary/. Read them, edit them, diff them — your editor is the admin UI.
2. No unreadable truth on disk. Every derived artifact (indexes, vector caches) is deletable and rebuildable from the files. The BM25 index lives in memory, built at startup.
3. Never block the conversation. Retrieval is synchronous, budgeted, and fail-open (0 LLM calls); memorization is async (≤ 1 LLM call by the host).
4. What happened is always answerable. Every mutation appends one line to journal.jsonl; retrieval can explain its scoring.

Wiki-links: what and why

A wiki-link is an in-content reference — the [[...]] syntax you may know from wikis and Obsidian — and in wikimem it always points at one item:

# preferences.md
## likes-the-sea

喜欢海边，提到过想去海边玩。[[daily_life:beach-trip-plan]]

<!-- wikimem: owner=user:xnne | source=conv_20260710 | ts=2026-07-10T03:00:00+00:00 -->

# daily_life.md
## beach-trip-plan

计划夏天去海边旅行，看日出。

[[daily_life:beach-trip-plan]] is an address with two parts: RecallFile (which file — daily_life.md) and item name (which ## heading inside it). So the linked node is an item: a named, self-contained entry of a few sentences — not a word, and not a whole file. When retrieval hits likes-the-sea, it mechanically expands its links one hop and injects the whole beach-trip-plan item alongside — no LLM call, no graph database; the "graph" is just text, and expansion is an exact-name lookup.

Why links, when there's already search?

• Search finds similar wording; links encode related meaning. A coffee preference and a morning routine may share no words — no keyword (often not even embedding) match connects them. A link written at memorization time does.
• One unit everywhere. The link target is the same unit retrieval ranks and the token budget trims: an item. Finer-grained than Obsidian's file-sized notes, so expanding a link never dumps an entire document into the prompt.
• Readable and writable by everyone. The extraction LLM emits links in the same single pass that writes the memory; you can add or fix them in any text editor; git diff shows them.
• Zero infrastructure, fail-soft. This replaces a graph database (the design it supersedes ran Neo4j for exactly this). A dangling link — target renamed or deleted — is tolerated and reported, never a crash.

Status

Pre-alpha, built milestone by milestone (design: XnneHangLab ADR-0001 — memory pipeline):

• M1 ✅ — storage layer: RecallFiles, item model + metadata, wiki-link parsing, journal.jsonl, atomic writes
• M2 ✅ — retrieval: in-memory BM25 (char-bigram fallback, [zh] extra for jieba), one-hop wiki-link expansion, token budget, explain
• M3 ✅ — optional embedding fusion ([embed] extra): content-hash vector cache (versioned .npy + plain-text keys), memmap tiers with binary quantization above 10k items, pluggable VectorIndex port, silent BM25 fallback when the endpoint is down
• M4 ✅ — CLI: ls / show / grep / explain / graph
• M5 ✅ — diary + time gate: append-only diary/YYYY-MM-DD.md event files, a stdlib parser for relative time expressions, and one ranked list where diary and wiki items compete on the same scale
• M6 — serve (HTTP + JSON thin shell): designed, deliberately not built yet

Past M5 the work is tracked per decision, not per milestone — see the ADR index, where every decision carries its own implementation status.

Quick start

from wikimem import MemoryIndex, MemoryStore

store = MemoryStore("memory/")
store.add("preferences", "likes-the-sea",
          "喜欢海边，提到过想去海边玩。[[daily_life:beach-trip-plan]]",
          owner="user:xnne", source_conv="conv_20260710")
store.add("daily_life", "beach-trip-plan", "计划夏天去海边旅行，看日出。")

index = MemoryIndex(store)  # in-memory BM25, rebuilds itself on store writes
result = index.retrieve("想去海边玩", budget_tokens=800)
for entry in result.items:
    # hits come ranked; each is followed by its one-hop wiki-link targets
    print(entry.source, entry.item.name, entry.score, entry.matched_terms)

Retrieval makes zero LLM calls and never persists the BM25 index — delete nothing, lose nothing. Install wikimem[zh] for jieba-based Chinese tokenization (default is character bigrams).

Optional semantic fusion (pip install wikimem[embed]) — BM25 is never disabled: with an embedder, every query runs both signals and fuses them (each min-max normalized — the same hybrid formula memU ADR-0007 converged on). BM25 catches the wording, cosine catches the meaning:

from wikimem.vectors import HttpEmbedder

embedder = HttpEmbedder("https://api.example.com/v1", "bge-m3", api_key="sk-…")
index = MemoryIndex(store, embedder=embedder)
result = index.retrieve("海滨度假")   # finds 喜欢海边 even with zero shared words
print(result.embedding_used)          # False = endpoint was down, BM25 carried on

Vectors live in a content-hash cache next to your markdown (versioned vectors-*.npy + readable vectors.keys.jsonl) — incrementally updated, deletable anytime, never the source of truth. An unreachable embedding endpoint silently degrades retrieval to BM25-only; it never raises.

CLI

The same store, from the shell — stdlib only, ships with the package:

wikimem -s memory/ ls                      # RecallFiles + item counts
wikimem -s memory/ show preferences        # print a RecallFile as stored
wikimem -s memory/ grep 海边                # regex search, grep-style output
wikimem -s memory/ explain "想去海边玩"      # retrieval scoring breakdown
wikimem -s memory/ graph --format mermaid  # wiki-link graph (mermaid / json)

graph takes over the retired Neo4j semantic-layer visualization: the same relation picture, parsed straight out of the markdown. Full reference: https://wikimem.xnnehang.top/reference/cli.

Development

uv sync
uv run pytest

Apache-2.0. Extraction-prompt design borrows from memU (Apache-2.0) — see lab ADR-0002.