Skip to content

feat: add LodeDB vector store - #847

Draft
idas0 wants to merge 1 commit into
Cinnamon:mainfrom
idas0:feat/lodedb-vector-store
Draft

feat: add LodeDB vector store#847
idas0 wants to merge 1 commit into
Cinnamon:mainfrom
idas0:feat/lodedb-vector-store

Conversation

@idas0

@idas0 idas0 commented Jul 12, 2026

Copy link
Copy Markdown

Description

Draft — this depends on the upcoming lodedb release that adds the kotaemon adapter
(tracking Egoist-Machines/LodeDB#82). Once that release is on PyPI, pip install lodedb
pulls the adapter and this moves to ready-for-review. The adapter is not in the current
published lodedb yet.

Adds kotaemon.storages.LodeDBVectorStore, a new optional vector-store backend, following
the same pattern as the existing Milvus and Qdrant additions: one wrapper module, exports, a
commented flowsettings.py line, an optional dependency in the adv extra, and a test class
mirroring TestChromaVectorStore.

LodeDB (Apache-2.0) is a local-first embedded
vector database — in-process, on-disk, no server or daemon — which matches kotaemon's local,
private deployment model. Two properties are a good fit for kotaemon's file-index workload:

  • Exact retrieval. The default is a brute-force exact scan, so the true nearest chunk is
    never missed by an index structure, regardless of collection size.
  • Incremental persistence. Commits write only changed rows, so repeated file uploads into
    a large index stay fast (no index rebuild).

Numbers (vs the current Chroma default)

Measured through kotaemon's own BaseVectorStore interface (the path ktem's file index
drives), one process per backend with byte-identical inputs on both sides: 20k chunks ×
384-dim normalized vectors, added in batches of 100, on an Apple-silicon CPU. Top-1 accuracy
is scored against a brute-force cosine oracle; the benchmark script is available on request.

metric LodeDB Chroma (default)
ingest 20k chunks 1.8 s 16.9 s
per-100-chunk add (durable commit) p50 / p95 / p99 8.6 / 8.9 / 9.0 ms 83.9 / 111 / 126 ms
query p50 / p95 (top-10) 0.17 / 0.22 ms 64.2 / 65.1 ms
scoped query p50 (500-chunk doc_ids scope) 0.31 ms 64.5 ms
top-1 accuracy vs brute-force oracle 1.000 0.440
disk 10.5 MB 424 MB
peak RSS 592 MB 577 MB
cold reopen + first query 0.43 s 0.07 s

What's in the diff (175 insertions, no behavior change unless selected)

  • libs/kotaemon/kotaemon/storages/vectorstores/lodedb.py — thin wrapper; lodedb is
    imported lazily and raises the conventional install hint when missing.
  • exports in storages/__init__.py and vectorstores/__init__.py.
  • one commented line in flowsettings.py:
    # "__type__": "kotaemon.storages.LodeDBVectorStore", # pip install lodedb
  • lodedb added to the adv optional extra.
  • TestLodeDBVectorStore in tests/test_vectorstore.py, mirroring the Chroma tests,
    skipif-guarded so CI without lodedb skips it cleanly.

The heavy lifting lives on the LodeDB side (lodedb.local.integrations.kotaemon), which
implements the BaseVectorStore contract against kotaemon's real usage: doc_ids chunk
scopes and MetadataFilters push down into LodeDB's metadata planner as exact filters, the
embedding dimension is discovered from the first add (kotaemon never configures one), and
MMR hints are accepted for drop-in parity. It deliberately imports nothing from kotaemon, so
there is no version coupling in either direction.

How this was tested

  • pytest libs/kotaemon/tests/test_vectorstore.py -k "LodeDB or Chroma" — 10 passed.
  • The full new-backend path via theflow deserialize
    ({"__type__": "kotaemon.storages.LodeDBVectorStore", ...}), real
    DocumentWithEmbedding / MetadataFilters objects.
  • black 22.3.0, flake8 (max-line-length 88, E203 ignored), isort --profile black all
    clean on the changed files.

Adds kotaemon.storages.LodeDBVectorStore, a thin wrapper over LodeDB's
kotaemon adapter (local-first embedded vector DB: in-process, on-disk,
exact-scan retrieval, incremental O(changed) persistence). Selectable via
the usual flowsettings switch; lodedb is an optional dependency (adv extra)
and the wrapper raises the standard install hint when it is missing.

Includes a TestLodeDBVectorStore suite mirroring TestChromaVectorStore
(skipped when lodedb is not installed).
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant