Conversation
zackcam
force-pushed
the
topk-pr117-rebuild
branch
from
September 16, 2026 18:50
aa5eb2e to
1076827
Compare
Vendors upstream heavykeeper-rs at tag v0.7.1 (commit a5191aa, tip merge d410785) as a local path dependency so it can be modified in-tree. This baseline already includes upstream's native byte serialization for CuckooTopK and BucketedTopK, the shared serialization module with magic bytes, a variant tag, an on-load hasher probe, RDB-format hardening, and the priority-queue redundant-lookup perf fix. Provenance: https://github.com/pmcgleenon/heavykeeper-rs at v0.7.1 Signed-off-by: Cameron Signed-off-by: Cameron Zack <zackcam@amazon.com>
Remove files not needed by the vendored crate build: the data/ test corpus (war_and_peace.txt plus its generator scripts and README), the examples/ programs (basic, ip_files, word_count) which exist only to demo the standalone crate and read the removed data corpus, and the upstream .github/ CI config (dependabot, release-plz, rust workflows). These are development and CI artifacts of the standalone heavykeeper-rs repo and play no part in building the module. The Rust source, benches, LICENSE, LICENSE-APACHE, README, CHANGELOG, and MODIFICATIONS.md are kept so the vendored copy stays a recognizable upstream snapshot. Signed-off-by: Cameron Zack <zackcam@amazon.com>
Signed-off-by: Tracy <yuningt@amazon.com> Signed-off-by: Cameron Zack <zackcam@amazon.com>
Introduce Fingerprint and Counter traits with impls for u64/u32/u16, and parameterize CuckooTopK/CuckooBuilder over the cell storage widths. Both default to u64, so existing CuckooTopK<T> usage is unchanged. The decay threshold lookup table is retained. Signed-off-by: Tracy <yuningt@amazon.com> Signed-off-by: Cameron Zack <zackcam@amazon.com>
Signed-off-by: Tracy <yuningt@amazon.com> Signed-off-by: Cameron Zack <zackcam@amazon.com>
Signed-off-by: Tracy <yuningt@amazon.com> Signed-off-by: Cameron Zack <zackcam@amazon.com>
Signed-off-by: Tracy <yuningt@amazon.com> Signed-off-by: Cameron Zack <zackcam@amazon.com>
Signed-off-by: Tracy <yuningt@amazon.com> Signed-off-by: Cameron Zack <zackcam@amazon.com>
…ions Signed-off-by: Cameron Zack <zackcam@amazon.com>
Signed-off-by: Cameron Zack <zackcam@amazon.com>
Signed-off-by: Cameron Zack <zackcam@amazon.com>
Signed-off-by: Cameron Zack <zackcam@amazon.com>
zackcam
force-pushed
the
topk-pr117-rebuild
branch
from
September 16, 2026 18:51
1076827 to
8f01e37
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This is a rebuild of: #117 with different changes. Thank you detemmienation for setting this up!
What this does
This change vendors the heavykeeper crate into the repo as a local path dependency and tunes it for Valkey. Before this, the module pulled heavykeeper from an external git branch. Vendoring it lets us make TopK-specific memory and correctness changes without waiting on the upstream crate.
Main changes
Vendoring and build:
Memory reductions (the TopK sketch uses CuckooTopK):
Correctness and upkeep:
Tests
Performance and memory
two example tables, at large large k and sketches, we cap at 15% for more memory (Note this is using the items added for redis memory which is untracked by memory usage in redis but is a part of the topk object)
Notes
The vendored crate keeps upstream's other implementations (TopK, BucketedTopK) even though the module only uses CuckooTopK, to stay a faithful mirror of the upstream crate.
For the future we will try and get these changes accepted upstream but if unable we will continue to use the vendored crate