feat(store): add createRwFile API and self-contained rw metadata management - #445
Draft
kisielewski wants to merge 20 commits into
Draft
feat(store): add createRwFile API and self-contained rw metadata management#445kisielewski wants to merge 20 commits into
kisielewski wants to merge 20 commits into
Conversation
Introduces CacheInterface with InMemory and NoOp implementations, plus a temporary GlobalCache singleton.
…ation Refactor FileHandler to accumulate writes and truncates into an in-memory dirty-chunk map instead of uploading to the server on every write() call. Dirty chunks are encrypted and sent in a single batch on flush()/close(), reducing round-trips for workloads that perform many small random writes (e.g. SQLite index updates via the Search API VFS layer). Introduce a per-connection scoped chunk read cache backed by GlobalCache: - CacheScopedNamespace wraps a shared CacheInterface and prepends a "host;pubkey;" prefix to every key, isolating cache entries between different Bridge connections and users without separate instances. - CacheKey provides a factory for cache key strings with a consistent "type;entity_id;sub_key" format. - GlobalCache is refactored: lazy thread-safe init via std::once_flag, new setChunksCache() / setChunksCacheEnabled() API for runtime configuration before the first connect(), renamed getChunksCacheInstance(). - ChunkDataProvider populates the cache for every chunk in a fetched server segment on first read, validates hits by HMAC, and evicts on mismatch. Freshly written chunks are also inserted into the cache after upload. Propagate the scoped cache instance through FileReadHandle and FileReadWriteHandle constructors and expose it to InboxApiImpl so inbox file reads share the same cache infrastructure. Expose flushFile() and getFileSizeFromHandle() on StoreApi / StoreApiImpl to allow callers to trigger a batch flush and query pending file size without a server round-trip. Update PrivmxFS (Search API): sync() calls flushFile(), getFileSize() reads from the open handle, and access() blocks WAL file paths to prevent SQLite from attempting write-ahead logging against the PrivMX backend.
…e-optimization-for-search-api
- Make KvdbApiImpl::createKvdbEx public so SearchApiImpl can call it - Fix DynamicTypes.hpp in search module: replace missing privmx/endpoint/core/TypesMacros.hpp with the correct privmx/utils/TypedObject.hpp + privmx/utils/TypesMacros.hpp, and replace undefined ENDPOINT_CLIENT_TYPE with DECLARE_CORE_TYPE
flush() splits chunks into batches sized upfront from the fixed encrypted chunk size; plaintext for gap chunks is allocated one chunk at a time to avoid N×plainChunkSize spikes. Each batch gets its own encrypted meta and advances local state immediately, so a failure in a later batch leaves the client consistent with what the server has already committed.
…dings Writes go directly to storeApi; Writer class removed. VFS callbacks use nullptr for unused params and structs are completed with version 2/3 slots.
Throw InvalidFileReadWriteHandleException instead of silently no-op or dereferencing null. Rename getFileSizeFromHandle -> getFileSize.
…-read-write-optimization-for-search-api
Replace three inline-duplicated expressions in FileHandler with named private helpers: committedChunkCount(), isMidTruncateActive(), and midTruncateClearStart(). Also promote PendingChunk from a flush()-local struct to a private nested type in the class.
…gement Introduce createRwFile() — a new public API method that creates an empty random-write file directly, without streaming chunks. The file is ready for read/write access via openFile immediately after creation. Refactor FileHandler to own its dynamic metadata entirely: replace the FileInfo/FileMeta/DecryptedEncKey/FileMetaEncryptor dependencies with a lightweight StaticMeta struct (fileId, resourceId, cipherType, chunkSize, key). FileHandler now computes and signs the rwMeta JSON on every flush using its own file key, and pulls the current server state via storeFileRwPull instead of receiving it as parameters. Simplify FileHandler::sync() to take no arguments — it fetches fresh rw state from the server autonomously. Simplify updateOnServer() to accept only newEncryptedFileSize; it builds and signs the full rwMeta internally. FileReadWriteHandle now calls storeFileRwPull on open to bootstrap the rw state, replacing the previous reliance on pre-fetched EncryptionParams. Server API: add storeFileRwPull, storeFileRwCreate, storeFileRwWrite methods with updated struct names (StoreFileRw* convention) and field names matching the bridge (rwVersion, rwMeta).
kisielewski
commented
Jun 10, 2026
| dmForSigning->set("serverSize", dmServerSize); | ||
| dmForSigning->set("size", dmSize); | ||
| dmForSigning->set("version", newVersion); | ||
| std::string rawSignature = privmx::crypto::Crypto::hmacSha256( |
Member
Author
There was a problem hiding this comment.
JSON serialization is not deterministic across libraries — key ordering, whitespace, and encoding of special values can differ between Poco (C++), and bindings in JS, Java, Swift, C#. Since this string is the HMAC input, any subtle difference breaks signature verification on the other end.
Worth considering a deterministic format, or simply signing a manually constructed string with fields in a fixed order (e.g. size || serverSize || version || hmac).
Base automatically changed from
feat/chunk-read-write-optimization-for-search-api
to
feat/search-api
July 30, 2026 06:01
Uriagat
force-pushed
the
feat/search-api
branch
2 times, most recently
from
August 4, 2026 10:06
b45c7d4 to
75bbe2f
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Important
Depends on:
Summary
StoreApi::createRwFile()— creates an empty random-write file directly, without streaming chunks; ready for read/write access viaopenFileimmediately after creation.FileHandlerto own its dynamic metadata entirely — replaceFileInfo/FileMeta/DecryptedEncKey/FileMetaEncryptordependencies with a lightweightStaticMetastruct.FileHandlernow computes and signsrwMetaJSON on every flush using its own file key.FileHandler::sync()to take no arguments — fetches current rw state from the server autonomously viastoreFileRwPull.FileHandler::updateOnServer()— accepts onlynewEncryptedFileSize; builds and signs the fullrwMetainternally.FileReadWriteHandlenow callsstoreFileRwPullon open to bootstrap state, replacing reliance on pre-fetchedEncryptionParams.StoreRwFile*→StoreFileRw*, fieldsversion/meta→rwVersion/rwMetainStoreFileRandomWriteMeta,StoreFileRwPullResultextended with optionalstoreandfilefields,StoreFileReadModelextended with optionalrwVersion.