Feat/search api - #472
Open
Uriagat wants to merge 62 commits into
Open
Conversation
Replaced manual string construction from istreambuf_iterator with Poco::StreamCopier::copyToString for improved readability and reliability.
Reserve string capacity based on response content length to improve performance and reduce reallocations when reading the HTTP response.
Modified file corruption check to skip exception when original file size is zero, preventing false positives for empty files.
Adjusted stopIndex computation to correctly handle zero-length data, preventing out-of-range chunk index calculation.
Modified UpdateChunkData calculation to use std::max when truncating, ensuring that size deltas do not become negative for partially filled chunks.
Moved squashedChanges reference inside the loop to avoid overwriting it incorrectly when pushing new UpdateChanges, ensuring proper merging of consecutive chunks.
Removed file size bounds verification in seekg and seekp to allow seeking beyond current file size, deferring validation to later operations.
mplemented a function to sanitize file paths so that they comply with
the Bridge server requirements. The function replaces unsupported
characters with allowed ones ('/', '_', ':', '-') and removes whitespaces.
Handled the case where opening the SQLite database file fails, preventing a segmentation fault. Additionally, fixed a memory leak that occurred when the database file was opened successfully, ensuring proper resource cleanup in all cases.
…n-existent document from the search index
…n-existent document in the search index
* test: e2e testing for StreamApi * test: fix StreamApi * test: fix: e2e_runner * test: new Dataset * test: searchApi * fix: searchApi reading kvdbEntry * chore: cleanup * test: searchApi new tests * test: fix searchApi * test: Dataset update * test: Dataset Update
* fix: searchApi loop when kvdb data decrypt failed * fix: small fix + more logegrs
…m-helpers-programs
…ile size (#436) * feat: add cache abstraction layer to store module Introduces CacheInterface with InMemory and NoOp implementations, plus a temporary GlobalCache singleton. * feat(store,search): buffered writes, chunk read cache, and cache isolation Refactor FileHandler to accumulate writes and truncates into an in-memory dirty-chunk map instead of uploading to the server on every write() call. Dirty chunks are encrypted and sent in a single batch on flush()/close(), reducing round-trips for workloads that perform many small random writes (e.g. SQLite index updates via the Search API VFS layer). Introduce a per-connection scoped chunk read cache backed by GlobalCache: - CacheScopedNamespace wraps a shared CacheInterface and prepends a "host;pubkey;" prefix to every key, isolating cache entries between different Bridge connections and users without separate instances. - CacheKey provides a factory for cache key strings with a consistent "type;entity_id;sub_key" format. - GlobalCache is refactored: lazy thread-safe init via std::once_flag, new setChunksCache() / setChunksCacheEnabled() API for runtime configuration before the first connect(), renamed getChunksCacheInstance(). - ChunkDataProvider populates the cache for every chunk in a fetched server segment on first read, validates hits by HMAC, and evicts on mismatch. Freshly written chunks are also inserted into the cache after upload. Propagate the scoped cache instance through FileReadHandle and FileReadWriteHandle constructors and expose it to InboxApiImpl so inbox file reads share the same cache infrastructure. Expose flushFile() and getFileSizeFromHandle() on StoreApi / StoreApiImpl to allow callers to trigger a batch flush and query pending file size without a server round-trip. Update PrivmxFS (Search API): sync() calls flushFile(), getFileSize() reads from the open handle, and access() blocks WAL file paths to prevent SQLite from attempting write-ahead logging against the PrivMX backend. * fix(search,kvdb): fix build errors after devel merge - Make KvdbApiImpl::createKvdbEx public so SearchApiImpl can call it - Fix DynamicTypes.hpp in search module: replace missing privmx/endpoint/core/TypesMacros.hpp with the correct privmx/utils/TypedObject.hpp + privmx/utils/TypesMacros.hpp, and replace undefined ENDPOINT_CLIENT_TYPE with DECLARE_CORE_TYPE * feat(store): batch server writes with 5 MB limit per updateOnServer call flush() splits chunks into batches sized upfront from the fixed encrypted chunk size; plaintext for gap chunks is allocated one chunk at a time to avoid N×plainChunkSize spikes. Each batch gets its own encrypted meta and advances local state immediately, so a failure in a later batch leaves the client consistent with what the server has already committed. * refactor(search): remove Writer buffering and clean up SQLite VFS bindings Writes go directly to storeApi; Writer class removed. VFS callbacks use nullptr for unused params and structs are completed with version 2/3 slots. * fix(store): guard flushFile/getFileSize against invalid handles Throw InvalidFileReadWriteHandleException instead of silently no-op or dereferencing null. Rename getFileSizeFromHandle -> getFileSize. * fix(search): update getFileSizeFromHandle call to renamed getFileSize * style: apply clang-format to inbox, search, and store modules * fix(search): remove redundant sync() call before read in PrivmxFile * refactor(store): extract repeated FileHandler helpers Replace three inline-duplicated expressions in FileHandler with named private helpers: committedChunkCount(), isMidTruncateActive(), and midTruncateClearStart(). Also promote PendingChunk from a flush()-local struct to a private nested type in the class. * refactor(store): reorder FileHandler methods by visibility and call depth * docs(store): note thread-safety issues in getChunk() * refactor(store): rename server chunk to segment in ChunkDataProvider * refactor(store): replace Pson::BinaryString with core::Buffer in cache API * refactor(store): pass cache value by const ref in CacheInterface::put * refactor(store): move CacheInterface and GlobalCache to include_pub * test(store): add flushFile calls before read assertions in random write tests --------- Co-authored-by: Jerzy Czarkowski <jczarkowski@simplito.com>
…issing key (#439) Adds KvdbApi::findEntry backed by a new kvdbEntryFind server call that returns std::optional<KvdbEntry> when the key is absent, instead of throwing KvdbEntryDoesNotExistException. Updates PrivmxFS::access and getFileId in the search module to use findEntry, removing the try/catch exception-based control flow used previously to handle legitimately missing entries. Co-authored-by: Jerzy Czarkowski <jczarkowski@simplito.com>
Adds beginTransaction, commit, and rollback methods to FullTextSearch, SearchApiImpl, SearchApi, and SearchApiVarInterface (methods 14–16), allowing callers to batch document mutations inside an explicit SQLite transaction. Adds matching TransactionBeginException, TransactionCommitException, and TransactionRollbackException (codes 0x0301–0x0303).
* feat: add cache abstraction layer to store module Introduces CacheInterface with InMemory and NoOp implementations, plus a temporary GlobalCache singleton. * feat(store,search): buffered writes, chunk read cache, and cache isolation Refactor FileHandler to accumulate writes and truncates into an in-memory dirty-chunk map instead of uploading to the server on every write() call. Dirty chunks are encrypted and sent in a single batch on flush()/close(), reducing round-trips for workloads that perform many small random writes (e.g. SQLite index updates via the Search API VFS layer). Introduce a per-connection scoped chunk read cache backed by GlobalCache: - CacheScopedNamespace wraps a shared CacheInterface and prepends a "host;pubkey;" prefix to every key, isolating cache entries between different Bridge connections and users without separate instances. - CacheKey provides a factory for cache key strings with a consistent "type;entity_id;sub_key" format. - GlobalCache is refactored: lazy thread-safe init via std::once_flag, new setChunksCache() / setChunksCacheEnabled() API for runtime configuration before the first connect(), renamed getChunksCacheInstance(). - ChunkDataProvider populates the cache for every chunk in a fetched server segment on first read, validates hits by HMAC, and evicts on mismatch. Freshly written chunks are also inserted into the cache after upload. Propagate the scoped cache instance through FileReadHandle and FileReadWriteHandle constructors and expose it to InboxApiImpl so inbox file reads share the same cache infrastructure. Expose flushFile() and getFileSizeFromHandle() on StoreApi / StoreApiImpl to allow callers to trigger a batch flush and query pending file size without a server round-trip. Update PrivmxFS (Search API): sync() calls flushFile(), getFileSize() reads from the open handle, and access() blocks WAL file paths to prevent SQLite from attempting write-ahead logging against the PrivMX backend. * fix(search,kvdb): fix build errors after devel merge - Make KvdbApiImpl::createKvdbEx public so SearchApiImpl can call it - Fix DynamicTypes.hpp in search module: replace missing privmx/endpoint/core/TypesMacros.hpp with the correct privmx/utils/TypedObject.hpp + privmx/utils/TypesMacros.hpp, and replace undefined ENDPOINT_CLIENT_TYPE with DECLARE_CORE_TYPE * feat(store): batch server writes with 5 MB limit per updateOnServer call flush() splits chunks into batches sized upfront from the fixed encrypted chunk size; plaintext for gap chunks is allocated one chunk at a time to avoid N×plainChunkSize spikes. Each batch gets its own encrypted meta and advances local state immediately, so a failure in a later batch leaves the client consistent with what the server has already committed. * refactor(search): remove Writer buffering and clean up SQLite VFS bindings Writes go directly to storeApi; Writer class removed. VFS callbacks use nullptr for unused params and structs are completed with version 2/3 slots. * fix(store): guard flushFile/getFileSize against invalid handles Throw InvalidFileReadWriteHandleException instead of silently no-op or dereferencing null. Rename getFileSizeFromHandle -> getFileSize. * fix(search): update getFileSizeFromHandle call to renamed getFileSize * style: apply clang-format to inbox, search, and store modules * fix(search): remove redundant sync() call before read in PrivmxFile * refactor(store): extract repeated FileHandler helpers Replace three inline-duplicated expressions in FileHandler with named private helpers: committedChunkCount(), isMidTruncateActive(), and midTruncateClearStart(). Also promote PendingChunk from a flush()-local struct to a private nested type in the class. * refactor(store): reorder FileHandler methods by visibility and call depth * docs(store): note thread-safety issues in getChunk() * refactor(store): rename server chunk to segment in ChunkDataProvider * refactor(store): replace Pson::BinaryString with core::Buffer in cache API * refactor(store): pass cache value by const ref in CacheInterface::put * refactor(store): move CacheInterface and GlobalCache to include_pub * test(store): add flushFile calls before read assertions in random write tests * feat(store): add SLRU-backed persistent chunk cache Introduces a segmented LRU (SLRU) eviction policy and a pluggable cache backend abstraction. GlobalCache now uses SlruBackedCache with an in-memory backend by default, and exposes setChunksCacheBackend() for external persistent backends (e.g. LevelDB). --------- Co-authored-by: Jerzy Czarkowski <jczarkowski@simplito.com>
SQLite queries the VFS for WAL file existence on every read/write operation. Since the Search module does not use WAL mode, intercepting these calls in PrivmxExtFS::access() avoids unnecessary round-trips to the server. Co-authored-by: Jerzy Czarkowski <jczarkowski@simplito.com>
* feat(lock): add dedicated Lock API module for resource locking Introduces a new lock module (endpoint/lock/) with lock, unlock, and checkReservedLock operations backed by direct server-side RPC calls. Replaces the KVDB-based LockSession/LockSetLogic in SearchApi, which required a read followed by a write — creating a TOCTOU window and doubling the number of bridge round-trips per lock acquisition. * fix(search): remove unused LockSession include from PrivmxFS.hpp --------- Co-authored-by: Jerzy Czarkowski <jczarkowski@simplito.com>
Uriagat
marked this pull request as ready for review
August 5, 2026 06:12
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.