docs: add design doc for replication fanout (NOT FOR MERGE) - #8002
docs: add design doc for replication fanout (NOT FOR MERGE)#8002BorysTheDev wants to merge 1 commit into
Conversation
| Every replica has its own connection, output queue, acknowledgement state, and progress. The | ||
| snapshot data and journal records are produced once per source shard, then distributed to those | ||
| independent flows. | ||
|
|
||
| ## Batching requests | ||
|
|
||
| At most one batch can create or send a snapshot. A request belongs either to the current batch or | ||
| to the next batch: |
There was a problem hiding this comment.
Please add an introduction what a batch is
| +--------------------------------------------+ | ||
| | | ||
| Serving node fanout | ||
| +----------------+ | | ||
| | snapshot maker |-------------------------------------------+-------+-------+ | ||
| +----------------+ | | | | ||
| v v v | ||
| R1 R2 R3 |
There was a problem hiding this comment.
With the current souce code you can have easy fan out only when you hide the three replicas behind a common interface
There was a problem hiding this comment.
The complexity question is: do we need batching or can we just handle everything one by one
|
Configuration: 4 GiB of random data, one proactor per instance, continuous 1 KiB
Fanout starts to improve full-sync time at three replicas with this collection-window setting. |
|
Configuration: requested data size 100.00 GiB, one master, 1–10 replicas, 10 proactors per Dragonfly process, 10 random-command client threads, a 1-second fanout collection window, and io_uring (default; locked-memory limit: 185.64 GiB).
|
2 participants
This is a design doc for #7993.