Repository navigation
Use mimalloc with purging disabled in each e2e benchmark binary - #10490
Merged
Merged
Conversation
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
The end-to-end benchmark binaries (SQL via DataFusion and DuckDB, random access, compression, string, Lance) only ran with mimalloc implicitly, through a
#[global_allocator]declared in thevortex-benchlibrary. This makes the choice explicit in each benchmark binary, and stops mimalloc from purging freed memory back to the OS during a run: mimalloc v3 purges 1s after a free, and that work lands in the middle of measured queries.Changes
#[global_allocator] static GLOBAL: MiMalloc = MiMalloc;in themain.rsofcompress-bench,datafusion-bench,duckdb-bench,lance-bench,random-access-bench,string-bench, and invortex-bench'sdata-genbinary.vortex-bench/src/lib.rs, so the library no longer picks the allocator for whatever links it.vortex_bench::configure_allocator(), which sets mimalloc'spurge_delayoption to-1(never purge), and call it first thing in each of those binaries. This useslibmimalloc-syswith itsextendedfeature (addsctyto the lockfile).mimallocto each benchmark crate's dependencies and updateCargo.lock.Checks:
cargo clippy --locked --bins -D warningsforvortex-bench,compress-bench,random-access-bench,string-benchanddatafusion-bench; formatted with the pinned nightlyrustfmt.duckdb-benchandlance-benchcould not be built locally (DuckDB source download and thelance-encodingbuild script fail in this environment), so CI covers them.🤖 Generated with Claude Code
https://claude.ai/code/session_0122rGn9fTSV47KLv191MTHB