[Question] Disaggregated training (producer:consumer 1:7) ~3.2x slower than DP for Qwen3.6-35B-A3B — expected?

[junzhang-zj]

Checklist

• 1. If the issue you raised is not a feature but a question, please raise a discussion at https://github.com/sgl-project/SpecForge/discussions/new/choose Otherwise, it will be closed.
• 2. Please use English, otherwise it will be closed.

Motivation

I'm training a draft model for Qwen3.6-35B-A3B using SpecForge and observing a significant wall-clock time difference between the disaggregated topology and the original DP (data-parallel) approach. I'd like to understand whether this gap is expected, and if there are recommended tuning strategies.

Setup

Item	Value
Target model	Qwen3.6-35B-A3B
Dataset size	~1.3M samples
Epochs	6
Training method	DFlash (disaggregated, online)
Producer : Consumer ratio	1 : 7 (still experimenting with other ratios)
Hardware	8× GPU single node

Observation

Method	Estimated total time
Original DP (colocated)	~45 hours
Disaggregated (producer:consumer = 1:7)	~6 days (~144 hours)

The disaggregated approach is roughly 3.2× slower in wall-clock time compared to the original DP method for the same dataset and epoch count.

Questions

1. Is this magnitude of slowdown expected for the disaggregated topology on a single node with a 1:7 producer-to-consumer ratio?
2. What are the primary bottlenecks — is it the feature capture throughput on the producer side, the Mooncake transfer overhead, or consumer-side waiting/starvation?
3. Are there recommended producer:consumer ratios or runtime tuning knobs (e.g., producer_lease, in_flight_high_watermark, in_flight_low_watermark) that can significantly close this gap?
4. Would using multiple capture servers (repeating server_urls) or offline pre-captured features be the recommended path for this model size?
5. Any known scaling behavior or best-practice configs for Qwen3.6-35B-A3B disaggregated training?

Additional context

• I'm still experimenting with different producer:consumer ratios; 1:7 is the current setting.
• The DP baseline uses the same hardware and dataset, just with the colocated target-inference + trainer approach.
• Happy to provide logs, throughput stats, or config YAML if helpful.

[jiapingW]

Hi, you can increase the Producer: Consumer ratio. We also have profiling data available; if you examine the profiles, you will likely see the consumers waiting for the producers to generate hidden states. The optimal configuration we determined for both the A100 and H20 is 3:5—that is, three producers and five consumers. Although this entails some underutilization of producers, it is an acceptable trade-off.

[jiapingW]

Of course, the optimal ratio may vary depending on the hardware.