commit into RT table with double buffer cause wrong stats

[tomatolog]

Bug Description:

commits into RT table large bulk of data that has multiple replaces \ documents with the same id and cause save of multiple disk chunks make word stats in some disk chunks inaccurate. That also causes differences between call keywords and matching weights.

The difference from the reference at the single node is hard to debug. However if there is a cluster with multiple nodes and commits goes into single node (node2 for example) even after commits are finished all nodes returns different stats for multiple terms

mysql -h 127.0.0.1 -P 17306 -e "select count(*) from test where match('am'); show meta"
|     7845 |
| keyword[0]     | am    |
| docs[0]        | 7876  |
| hits[0]        | 8538  |

mysql -h 127.0.0.1 -P 17406 -e "select count(*) from test where match('am'); show meta"
|     7845 |
| keyword[0]     | am    |
| docs[0]        | 7879  |
| hits[0]        | 8542  |

mysql -h 127.0.0.1 -P 17506 -e "select count(*) from test where match('am'); show meta"
|     7845 |
| keyword[0]     | am    |
| docs[0]        | 7878  |
| hits[0]        | 8541  |

even result sets data are the same returned by the select query the same difference and for call keywords. Dump of the disk chunk dictionary also shows the same difference

indextool -c n1.conf --dumpdict n{1,2,3}/test/test.11.spi
--- n1
keyword,docs,hits,offset
am,7876,8538,507207
--- n2
am,7879,8542,507347
--- n3
am,7878,8541,507335

Related data to reproduce case located at ~/bin/rpl/ref-stat/case_couble_buf.zip.

• start 3 daemons

~/build/m_reld/src/searchd -c node1.conf
~/build/m_reld/src/searchd -c node2.conf
~/build/m_reld/src/searchd -c node3.conf

• post initial batch of data mysql -h0 -P 32106 < test1k.sql
• run the script bash load.sh that in loop posts data in parallel into node2 then after posting check for the call keywords output and md5sum of spi on all nodes
• script got stopped on any spi file difference

Manticore Search Version:

master

Operating System Version:

linux

Have you tried the latest development version?

Yes

Internal Checklist:

^{To be completed by the assignee. Check off tasks that have been completed or are not applicable.}

Details

• Implementation completed
• Tests developed
• Documentation updated
• Documentation reviewed

[tomatolog]

the related PR that fixes the issue #4065 the I sugest to delete double buffer feature from the RT table.
The double buffer is the feature that allows to collect some more data above the rt_mem_limit settings during the disk chunk save. That was done long time ago to allow user post some data during the save disk and finish its job if the user has not much data left. That small extra data collected at the double buffer. However if the user post a lot of data and hit the double buffer limit it put on hold after the disk chunk finishes its work,

And after that fix #4065 if user posts the data and hits the rt_mem_limit > cause the save of disk chunk all write operations put on hold and resumes only after save of disk chunk finished.

I see no uses for double buffer now as we has auto-flushing a RAM chunk feature and all data will be flushed on disk someway.

[tomatolog]

the issue that during double buffer collecting phase RT index allow to alter the RAM segments these are used for saving disk chunk data. These segments stays the constant however kill-lists at these segments got changed this way the dictionary stats and doc-lists and hit-lists got different between nodes and the result set that enumerate the all rows show stats that differs from dictionary stats.

That affects the match weights and call keywords output for the RT table, ie different nodes at the cluster could return different results set for default sorting as manual said

If nothing is specified, they are sorted by relevance, which is equivalent to ORDER BY weight() DESC

That is why I suggest to remove double buffer feature instead of try to fix it.

[tomatolog]

that also causes the issue latter described here #3596 not the duplicates but the inconsistent ranking and https://forum.manticoresearch.com/t/inconsistent-relevance-scores-across-cluster-nodes-with-custom-ranker/2335

[klirichek]

The idea of double-buffer is quite different from sphinx one.
There were 90% of rt-mem-limit + 10% for double buffer.

In manticore we have flexible limit, so it may even became triple-buffer (with rate 33%). And now it significantly increases speed of insertion, as adding new documents and saving disk chunk works in parallel, allow to utilize whole disk write bandwidth. So, removing it will cause inserts to stall x2 or even x3 slower, because instead of parallel insertion it will be sequence of portion insert + disk write.

Core of the problem is 'transparent' killing, when we const-cast current chunks/segments and apply kill-list immediately despite constant state. That produces many 'funny things', starting from non-atomic replaces (kill comes first, re-insert follows) and all other inconsistencies like one you try to solve, avoiding touch the real problem.

FUI, saving segments cares about kill. At first stage, we collect all the documents among all 'frozen' segments. As we iterate from serment to segment, from 1-st doc to the last - we just check every kill how it affects docs in current segment, in current doc. If killed doc is not reached yet - that is ok to kill it; at the moment we reach it during iteration, the state will be consistent. But if we already processed document, and it got killed - we store 'mirror' kill-list to be applied over finally saved disk chunk. And that is, again, transparent - current user see killed documents in current ram segments; future user will see new disk chunk with same killed documents applied.

[tomatolog]

I checked data posting perf master vs no_double buffer branch and see the difference

the load script

manticore-load \
--drop \
--port=32106 \
--batch-size=1000 \
--threads=5 \
--total=2000000 \
--init="CREATE TABLE test(id bigint, name text, type int)" \
--load="INSERT INTO test(id,name,type) VALUES(<increment>,'<text/10/100>',<int/1/100>)"

no double

Process 1 final statistics:
Init command: CREATE TABLE test(id bigint, name text, type int)
Load command: INSERT INTO test(id,name,type) VALUES(,'<text/10/100>',<int/1/100>)
Total time: 01:41
Total queries: 2000
Threads: 5
Batch size: 1000
Total docs: 2000000
Docs per sec avg: 19675
QPS avg: 28
QPS 1p: 1
QPS 5p: 1
QPS 95p: 39
QPS 99p: 39
Latency avg: 254.3 ms
Latency 50p: 155.0 ms
Latency 95p: 245.0 ms
Latency 99p: 3950.0 ms

master

Process 1 final statistics:
Init command: CREATE TABLE test(id bigint, name text, type int)
Load command: INSERT INTO test(id,name,type) VALUES(,'<text/10/100>',<int/1/100>)
Total time: 58.9s
Total queries: 2000
Threads: 5
Batch size: 1000
Total docs: 2000000
Docs per sec avg: 33949
QPS avg: 34
QPS 1p: 12
QPS 5p: 31
QPS 95p: 38
QPS 99p: 38
Latency avg: 147.1 ms
Latency 50p: 145.0 ms
Latency 95p: 185.0 ms
Latency 99p: 205.0 ms

seems the double buffer removal could also affects data posting perf. Then in double buffer case dictionary stats should be fixed somehow.

[klirichek]

The core cause of different stats is not the way how disk chunks are saved, but the way how killing documents implemented.
If you just set a bit means doc is 'killed' - dictionary is NOT changed, N of docs for each keyword is also not affected.
If one node then perform optimize and actually rebuild dictionary - it will get correct stat, because N of words after rebuild is correct. If another node still use previous version of chunk with old dict + kill-list - it will show differ stats.
Also, having different (several) chunks show differ weights. That is old and fundamental problem, and we run our test suite permanently with '--no-weights' because of it.

So, just presence of kill-list in current form is the 'dept' by itself, it alone causes different stats. The way you fill your chunks - either by timer (slow rate), either with double-buffer (high rate), either manually (flush ramchunk) doesn't matter at all as long as you have kill/replace in your flow.

For the correct result, kill should not just 'mark' a document as dead, but it either should be followed by 'compressing' target chunk (which is very slow), or - by making whole 'anti-chunk' - i.e. extract original content from docstore, and make another dictionary, 'kill-dictionary', and then use it for searching together with basic one, but with sign 'minus'. That is - word 'foo' has 3 docs; we killed a doc containing word 'foo' - it produced kill-dict, where word 'foo' has 1 doc. On real search for word 'foo' we will met 3 words minus 1 word from kill-dict, totally 2 words, and it will produce correct stat. In current case doc just marked as killed, and from dict word 'foo' has anyway 3 docs for stat.

[tomatolog]

Checked daemon from the paralel merge branch there also added option to rebuild dictionary on kill-list update

I set searchd.kill_dictionary = 1 options for all nodes config
Changed the load.sh script to compare the indextool --dumpdict output for each spi file and after 4 loops I see no difference reported by the indextool dumpdict comparison

The only issue that daemon slower on processing data compare to master

--- paralel merge
Threads ; Batch     ; Time        ; Total Docs  ; Docs/Sec    ; Avg QPS     ; p99 QPS     ; p95 QPS     ; p5 QPS      ; p1 QPS      ; Lat Avg     ; Lat p50     ; Lat p95     ; Lat p99     ;
5       ; 100       ; 00:18       ; 100000      ; 5547        ; 57          ; 85          ; 85          ; 14          ; 14          ; 89.8        ; 66.5        ; 175.0       ; 195.0       ;

--- master
Threads ; Batch     ; Time        ; Total Docs  ; Docs/Sec    ; Avg QPS     ; p99 QPS     ; p95 QPS     ; p5 QPS      ; p1 QPS      ; Lat Avg     ; Lat p50     ; Lat p95     ; Lat p99     ;
5       ; 100       ; 00:04       ; 100000      ; 23438       ; 236         ; 255         ; 255         ; 226         ; 226         ; 21.3        ; 18.5        ; 37.5        ; 45.5        ;

[dimimarinkovic]

Hi, is resolving this issue still on your roadmap? :)

[tomatolog]

no this is open issue that still has no resolution