kill_dictionary (WIP) by sanikolaev · Pull Request #4087 · manticoresoftware/manticoresearch

sanikolaev · 2025-12-30T11:34:35Z

This might fix #3596

- add settings `kill_dictionary` (0|realtime|flush|idle) and `kill_dictionary_idle_timeout` (config + `SET GLOBAL` + `SHOW VARIABLES`) - persist kill stats to `.spks` per chunk; load at preread; rebuild on flush/idle - expose `kill_dictionary_dirty_chunks` in `SHOW TABLE STATUS` for pending rebuilds - parallelize kill-stat builds and add profiling; update `indextool --dumpdict` columns and docs/tests - updated weights and counts in ubertests 044, 045, 057 and 068

…have to stop searchd when you know what you're doing

Print RAM-segment words using their stored length prefix instead of treating them as null-terminated C strings. Prevents control byte 0x03 from appearing in indextool --dumpdict output for RT tables with RAM chunks.

…viewing the PR)

github-actions · 2026-01-05T15:39:10Z

clt

❌ CLT tests in test/clt-tests/sharding/cluster/ test/clt-tests/sharding/functional/ test/clt-tests/sharding/replication/ test/clt-tests/sharding/syntax/
✅ OK: 18
❌ Failed: 1
⏳ Duration: 579s
👉 Check Action Results for commit 60c2cbc

Failed tests:

🔧 Edit failed tests in UI:

Edit test/clt-tests/sharding/replication/test-multi-node-sharding-and-replication.rec

test/clt-tests/sharding/replication/test-multi-node-sharding-and-replication.rec

––– input –––
export INSTANCE=1
––– output –––
OK
––– input –––
mkdir -p /var/{run,lib,log}/manticore-${INSTANCE}
––– output –––
OK
––– input –––
stdbuf -oL searchd -c test/clt-tests/base/searchd-with-flexible-ports.conf > /dev/null
––– output –––
OK
––– input –––
if timeout 10 grep -qm1 '\[BUDDY\] started' <(tail -n 1000 -f /var/log/manticore-${INSTANCE}/searchd.log); then echo 'Buddy started!'; else echo 'Timeout or failed!'; cat /var/log/manticore-${INSTANCE}/searchd.log; fi
––– output –––
OK
––– input –––
export INSTANCE=2
––– output –––
OK
––– input –––
mkdir -p /var/{run,lib,log}/manticore-${INSTANCE}
––– output –––
OK
––– input –––
stdbuf -oL searchd -c test/clt-tests/base/searchd-with-flexible-ports.conf > /dev/null
––– output –––
OK
––– input –––
if timeout 10 grep -qm1 '\[BUDDY\] started' <(tail -n 1000 -f /var/log/manticore-${INSTANCE}/searchd.log); then echo 'Buddy started!'; else echo 'Timeout or failed!'; cat /var/log/manticore-${INSTANCE}/searchd.log; fi
––– output –––
OK
––– input –––
export INSTANCE=3
––– output –––
OK
––– input –––
mkdir -p /var/{run,lib,log}/manticore-${INSTANCE}
––– output –––
OK
––– input –––
stdbuf -oL searchd -c test/clt-tests/base/searchd-with-flexible-ports.conf > /dev/null
––– output –––
OK
––– input –––
if timeout 10 grep -qm1 '\[BUDDY\] started' <(tail -n 1000 -f /var/log/manticore-${INSTANCE}/searchd.log); then echo 'Buddy started!'; else echo 'Timeout or failed!'; cat /var/log/manticore-${INSTANCE}/searchd.log; fi
––– output –––
OK
––– input –––
export INSTANCE=4
––– output –––
OK
––– input –––
mkdir -p /var/{run,lib,log}/manticore-${INSTANCE}
––– output –––
OK
––– input –––
stdbuf -oL searchd -c test/clt-tests/base/searchd-with-flexible-ports.conf > /dev/null
––– output –––
OK
––– input –––
if timeout 10 grep -qm1 '\[BUDDY\] started' <(tail -n 1000 -f /var/log/manticore-${INSTANCE}/searchd.log); then echo 'Buddy started!'; else echo 'Timeout or failed!'; cat /var/log/manticore-${INSTANCE}/searchd.log; fi
––– output –––
OK
––– input –––
export INSTANCE=5
––– output –––
OK
––– input –––
mkdir -p /var/{run,lib,log}/manticore-${INSTANCE}
––– output –––
OK
––– input –––
stdbuf -oL searchd -c test/clt-tests/base/searchd-with-flexible-ports.conf > /dev/null
––– output –––
OK
––– input –––
if timeout 10 grep -qm1 '\[BUDDY\] started' <(tail -n 1000 -f /var/log/manticore-${INSTANCE}/searchd.log); then echo 'Buddy started!'; else echo 'Timeout or failed!'; cat /var/log/manticore-${INSTANCE}/searchd.log; fi
––– output –––
OK
––– input –––
export CLUSTER_NAME=c
––– output –––
OK
––– input –––
mysql -h0 -P1306 -e "create cluster ${CLUSTER_NAME}"
––– output –––
OK
––– input –––
mysql -h0 -P1306 -e "show status like 'cluster_${CLUSTER_NAME}_status'\G"
––– output –––
OK
––– input –––
export CLUSTER_NAME=c
––– output –––
OK
––– input –––
for n in `seq 2 $INSTANCE`; do mysql -h0 -P${n}306 -e "join cluster ${CLUSTER_NAME} at '127.0.0.1:1312'"; done;
––– output –––
OK
––– input –––
mysql -h0 -P${INSTANCE}306 -e "show status like 'cluster_${CLUSTER_NAME}_status'\G"
––– output –––
OK
––– input –––
apt-get install -y jq 1> /dev/null 2>&1 ; echo $?
––– output –––
OK
––– input –––
mysql -h0 -P1306 -e "create table c:t(id bigint) shards='5' rf='3' timeout='300'" >/dev/null
––– output –––
+ ERROR 1064 (42000) at line 1: Waiting timeout exceeded.
––– input –––
grep 'started v' /var/log/manticore-{1,2,3,4,5}/searchd.log | cut -d\' -f3 | cut -d: -f3 | wc -l
––– output –––
OK
––– input –––
for i in 1 2 3 4 5; do echo "#{$i}"; mysql -h0 -P${i}306 -e "show create table t option force=1\G" | grep 'Create Table'; done
––– output –––
OK

github-actions · 2026-01-05T16:05:07Z

clt

❌ CLT tests in test/clt-tests/sharding/cluster/ test/clt-tests/sharding/functional/ test/clt-tests/sharding/replication/ test/clt-tests/sharding/syntax/
✅ OK: 18
❌ Failed: 1
⏳ Duration: 318s
👉 Check Action Results for commit 60c2cbc

Failed tests:

🔧 Edit failed tests in UI:

Edit test/clt-tests/sharding/functional/functional-sharding-positive.rec

test/clt-tests/sharding/functional/functional-sharding-positive.rec

––– input –––
export INSTANCE=1
––– output –––
OK
––– input –––
mkdir -p /var/{run,lib,log}/manticore-${INSTANCE}
––– output –––
OK
––– input –––
stdbuf -oL searchd -c test/clt-tests/base/searchd-with-flexible-ports.conf > /dev/null
––– output –––
OK
––– input –––
if timeout 10 grep -qm1 '\[BUDDY\] started' <(tail -n 1000 -f /var/log/manticore-${INSTANCE}/searchd.log); then echo 'Buddy started!'; else echo 'Timeout or failed!'; cat /var/log/manticore-${INSTANCE}/searchd.log; fi
––– output –––
OK
––– input –––
export INSTANCE=2
––– output –––
OK
––– input –––
mkdir -p /var/{run,lib,log}/manticore-${INSTANCE}
––– output –––
OK
––– input –––
stdbuf -oL searchd -c test/clt-tests/base/searchd-with-flexible-ports.conf > /dev/null
––– output –––
OK
––– input –––
if timeout 10 grep -qm1 '\[BUDDY\] started' <(tail -n 1000 -f /var/log/manticore-${INSTANCE}/searchd.log); then echo 'Buddy started!'; else echo 'Timeout or failed!'; cat /var/log/manticore-${INSTANCE}/searchd.log; fi
––– output –––
OK
––– input –––
export INSTANCE=3
––– output –––
OK
––– input –––
mkdir -p /var/{run,lib,log}/manticore-${INSTANCE}
––– output –––
OK
––– input –––
stdbuf -oL searchd -c test/clt-tests/base/searchd-with-flexible-ports.conf > /dev/null
––– output –––
OK
––– input –––
if timeout 10 grep -qm1 '\[BUDDY\] started' <(tail -n 1000 -f /var/log/manticore-${INSTANCE}/searchd.log); then echo 'Buddy started!'; else echo 'Timeout or failed!'; cat /var/log/manticore-${INSTANCE}/searchd.log; fi
––– output –––
OK
––– input –––
export CLUSTER_NAME=replication
––– output –––
OK
––– input –––
mysql -h0 -P1306 -e "create cluster ${CLUSTER_NAME}"
––– output –––
OK
––– input –––
mysql -h0 -P1306 -e "show status like 'cluster_${CLUSTER_NAME}_status'\G"
––– output –––
OK
––– input –––
for n in `seq 2 $INSTANCE`; do mysql -h0 -P${n}306 -e "join cluster ${CLUSTER_NAME} at '127.0.0.1:1312'"; done;
––– output –––
OK
––– input –––
mysql -h0 -P${INSTANCE}306 -e "show status like 'cluster_${CLUSTER_NAME}_status'\G"
––– output –––
OK
––– input –––
for port in 1306 2306 3306; do timeout 60 mysql -h0 -P$port -e "SHOW STATUS LIKE 'cluster_replication_status'\G" > /tmp/status_$port.log && grep -q "Value: primary" /tmp/status_$port.log && echo "Port $port: Node synced"; done && cat /tmp/status_1306.log
––– output –––
OK
––– input –––
mysql -h0 -P1306 -e "create table ${CLUSTER_NAME}:tbl1(id bigint) shards='5' rf='2'"; echo $?
––– output –––
OK
––– input –––
for i in 1 2 3; do mysql -h0 -P${i}306 -e "show tables from system\G"; done | grep 'system.tbl1_s' | sort -V
––– output –––
OK
––– input –––
mysql -h0 -P2306 -e "create table ${CLUSTER_NAME}:tbl2(id bigint) shards='4' rf='1'"; echo $?
––– output –––
OK
––– input –––
for i in 1 2 3; do mysql -h0 -P${i}306 -e "show tables from system\G"; done | grep 'system.tbl2_s' | sort -V
––– output –––
OK
––– input –––
mysql -h0 -P3306 -e "create table ${CLUSTER_NAME}:tbl3(id bigint) shards='5' rf='2'"; echo $?
––– output –––
OK
––– input –––
for i in 1 2 3; do mysql -h0 -P${i}306 -e "show tables from system\G"; done | grep 'system.tbl3_s' | sort -V
––– output –––
OK
––– input –––
mysql -h0 -P3306 -e "DROP TABLE ${CLUSTER_NAME}:tbl3"; echo $?
––– output –––
OK
––– input –––
mysql -h0 -P3306 -e "create table ${CLUSTER_NAME}:tbl3(id bigint) shards='10' rf='3'"; echo $?
––– output –––
- 0
+ ERROR 1064 (42000) at line 1: Waiting timeout exceeded.
+ 1
––– input –––
for i in 1 2 3; do mysql -h0 -P${i}306 -e "show tables from system\G"; done | grep 'system.tbl3_s' | sort -V
––– output –––
OK
––– input –––
mysql -h0 -P3306 -e "create table ${CLUSTER_NAME}:tbl4(id bigint) shards='3000' rf='3' timeout='60'"; echo $?
––– output –––
OK
––– input –––
for i in 1 2 3; do mysql -h0 -P${i}306 -e "show tables from system\G"; done | grep -c 'system.tbl4_s'
––– output –––
OK
––– input –––
mysql -h0 -P3306 -e "create table ${CLUSTER_NAME}:tbl5(id bigint) shards='3001' rf='3'"?
––– output –––
OK

- add queued pending-rows path for flush mode and hook kills to enqueue - derive kill-stats keys via FNV64 for keyword dicts and use them in dump/apply - write/read .spks with magic/version header and validate on load - reset pending rows on dealloc/build/load and add detailed flush update/write timing logs - added validation examples to the feature doc (kill-dictionary.md)

sanikolaev · 2026-03-24T05:21:39Z

@dimimarinkovic wants to help with testing

@tomatolog please merge master into this branch and make sure the package artifacts and Docker image are built (I added the 'pack' label to this PR, which should trigger building the packages and uploading them as workflow artifacts)

…d index

tomatolog · 2026-03-24T17:17:55Z

I merged master into this branch and
the packages got build at the https://github.com/manticoresoftware/manticoresearch/actions/runs/23499551655 however these are avalable only in the CI artifact page - not at the repo as the publishing step was skipped in the Github CI Action

CLAassistant · 2026-08-28T12:53:35Z

Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you all sign our Contributor License Agreement before we can accept your contribution.
2 out of 3 committers have signed the CLA.

✅ tomatolog
✅ sanikolaev
❌ github-actions[bot]
_{You have signed the CLA already but the status is still pending? Let us recheck it.}

sanikolaev force-pushed the kill_dictionary branch 3 times, most recently from 820e2a4 to f7ff92b Compare January 3, 2026 10:01

sanikolaev added 5 commits January 4, 2026 00:12

feat: add indextool --skip-lock for unsafe RT dumpdict, so you don't …

4ce5da8

…have to stop searchd when you know what you're doing

fix: indextool --dumpdict output for RT RAM segments

939ac22

Print RAM-segment words using their stored length prefix instead of treating them as null-terminated C strings. Prevents control byte 0x03 from appearing in indextool --dumpdict output for RT tables with RAM chunks.

chore: add mysql_word_counts CLI for word stats

ef68a82

chore: updated kill-dictionary.md (the file is to be removed after re…

b8fbc1a

…viewing the PR)

sanikolaev force-pushed the kill_dictionary branch from c48caf3 to b8fbc1a Compare January 3, 2026 17:15

manticoresoftware deleted a comment from github-actions Bot Jan 3, 2026

sanikolaev force-pushed the kill_dictionary branch from cf551b8 to 2b06aae Compare January 6, 2026 15:39

sanikolaev requested a review from tomatolog February 11, 2026 09:17

sanikolaev mentioned this pull request Mar 11, 2026

commit into RT table with double buffer cause wrong stats #4066

Open

4 tasks

sanikolaev added the pack To trigger CI to build packages for PR label Mar 24, 2026

tomatolog and others added 5 commits March 24, 2026 16:26

minimize changes to master; reverted auto-translation

a3e2e05

minimize changes to master; reverted ubertests and CLT changes

8eece4a

merged master into kill_dictionary

8657d02

docs: Auto-translate documentation changes by tomatolog

d88f8aa

fixed test 323 to set explicit order of the result set for distribute…

b5b56c4

…d index

Navigation Menu

Uh oh!

kill_dictionary (WIP) - #4087

kill_dictionary (WIP)#4087
sanikolaev wants to merge 11 commits into
old-masterfrom
kill_dictionary

sanikolaev commented Dec 30, 2025 •

edited

Loading

Uh oh!

github-actions Bot commented Jan 5, 2026

Uh oh!

github-actions Bot commented Jan 5, 2026

Uh oh!

sanikolaev commented Mar 24, 2026

Uh oh!

tomatolog commented Mar 24, 2026

Uh oh!

CLAassistant commented Aug 28, 2026 •

edited

Loading

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Uh oh!

Conversation

sanikolaev commented Dec 30, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions Bot commented Jan 5, 2026

clt

Failed tests:

Uh oh!

github-actions Bot commented Jan 5, 2026

clt

Failed tests:

Uh oh!

sanikolaev commented Mar 24, 2026

Uh oh!

tomatolog commented Mar 24, 2026

Uh oh!

CLAassistant commented Aug 28, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

sanikolaev commented Dec 30, 2025 •

edited

Loading

CLAassistant commented Aug 28, 2026 •

edited

Loading