feat: add LodeDB vector store - #847
Draft
idas0 wants to merge 1 commit into
Draft
Conversation
Adds kotaemon.storages.LodeDBVectorStore, a thin wrapper over LodeDB's kotaemon adapter (local-first embedded vector DB: in-process, on-disk, exact-scan retrieval, incremental O(changed) persistence). Selectable via the usual flowsettings switch; lodedb is an optional dependency (adv extra) and the wrapper raises the standard install hint when it is missing. Includes a TestLodeDBVectorStore suite mirroring TestChromaVectorStore (skipped when lodedb is not installed).
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Adds
kotaemon.storages.LodeDBVectorStore, a new optional vector-store backend, followingthe same pattern as the existing Milvus and Qdrant additions: one wrapper module, exports, a
commented
flowsettings.pyline, an optional dependency in theadvextra, and a test classmirroring
TestChromaVectorStore.LodeDB (Apache-2.0) is a local-first embedded
vector database — in-process, on-disk, no server or daemon — which matches kotaemon's local,
private deployment model. Two properties are a good fit for kotaemon's file-index workload:
never missed by an index structure, regardless of collection size.
a large index stay fast (no index rebuild).
Numbers (vs the current Chroma default)
Measured through kotaemon's own
BaseVectorStoreinterface (the pathktem's file indexdrives), one process per backend with byte-identical inputs on both sides: 20k chunks ×
384-dim normalized vectors, added in batches of 100, on an Apple-silicon CPU. Top-1 accuracy
is scored against a brute-force cosine oracle; the benchmark script is available on request.
doc_idsscope)What's in the diff (175 insertions, no behavior change unless selected)
libs/kotaemon/kotaemon/storages/vectorstores/lodedb.py— thin wrapper;lodedbisimported lazily and raises the conventional install hint when missing.
storages/__init__.pyandvectorstores/__init__.py.flowsettings.py:# "__type__": "kotaemon.storages.LodeDBVectorStore", # pip install lodedblodedbadded to theadvoptional extra.TestLodeDBVectorStoreintests/test_vectorstore.py, mirroring the Chroma tests,skipif-guarded so CI withoutlodedbskips it cleanly.The heavy lifting lives on the LodeDB side (
lodedb.local.integrations.kotaemon), whichimplements the
BaseVectorStorecontract against kotaemon's real usage:doc_idschunkscopes and
MetadataFilterspush down into LodeDB's metadata planner as exact filters, theembedding dimension is discovered from the first
add(kotaemon never configures one), andMMR hints are accepted for drop-in parity. It deliberately imports nothing from kotaemon, so
there is no version coupling in either direction.
How this was tested
pytest libs/kotaemon/tests/test_vectorstore.py -k "LodeDB or Chroma"— 10 passed.deserialize(
{"__type__": "kotaemon.storages.LodeDBVectorStore", ...}), realDocumentWithEmbedding/MetadataFiltersobjects.black22.3.0,flake8(max-line-length 88, E203 ignored),isort --profile blackallclean on the changed files.