Skip to content

fix: reduce infix merge memory usage - #4881

Closed
sanikolaev wants to merge 1 commit into
mainfrom
fix/4850-infix-merge-memory
Closed

sanikolaev wants to merge 1 commit into
mainfrom
fix/4850-infix-merge-memory

Conversation

@sanikolaev

Copy link
Copy Markdown
Collaborator
  • store the infix arena and long UTF-8 keys in bounded segments
  • compact short keys and one/two-checkpoint lists inline
  • sort arena blocks and serialize them with a k-way merge
  • honor min_infix_len throughout indexing and merge paths
  • release builder storage as soon as infix entries are written
  • add a local peak-RSS benchmark

Related issue: #4850

- store the infix arena and long UTF-8 keys in bounded segments
- compact short keys and one/two-checkpoint lists inline
- sort arena blocks and serialize them with a k-way merge
- honor min_infix_len throughout indexing and merge paths
- release builder storage as soon as infix entries are written
- add a local peak-RSS benchmark

Related issue: #4850
@sanikolaev

Copy link
Copy Markdown
Collaborator Author

Benchmarks show that this approach doesn't lead to significant improvements.

@sanikolaev sanikolaev closed this Sep 11, 2026
@sanikolaev

Copy link
Copy Markdown
Collaborator Author

Benchmarks show that this approach doesn't lead to significant improvements.

The command I tested with was:

sudo -u manticore manticore-load --cache-gen-workers=10   --wait   --cache-from-disk   --drop --port=9306   --threads=10   --batch-size=10000   --total=1600000   --init="CREATE TABLE t(title text) min_infix_len='3' index_exact_words='1' optimize_cutoff='1'; SET GLOBAL parallel_chunk_merges=1" --load="INSERT INTO t(id,title) VALUES(<increment>,'word<increment><string/24/24> word<increment><string/24/24> word<increment><string/24/24> word<increment><string/24/24> word<increment><string/24/24> word<increment><string/24/24> word<increment><string/24/24> word<increment><string/24/24>')"

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant