core is not reviewed yet, firstly glanced at the non-core things

only that tiny nit and then ready to merge

LGTM, now only need to wait for CI green

@zhyncs Could you view this pr, waiting approval to merge

@House-West Nice work! For the first case, I wonder to know where the performance gain comes from. Is two chunk overlap equivalent to two batch overlap under that case? Thanks.

@House-West Nice work! For the first case, I wonder to know where the performance gain comes from. Is two chunk overlap equivalent to two batch overlap under that case? Thanks.

@ch-wan as mentioned in #6328. When each dp has only one request, two-chunk-overlap and two- batch-overlap are not equivalent.
such as batch_size = 1, extend_seq_len = [3072], extend_prefix_len = [0]

In two- batch-overlap:

• micro batch0: extend_seq_len = [3072], extend_prefix_len = [0]
• micro batch1 (idle batch): extend_seq_len = [0], extend_prefix_len = [0]

In two- chunk-overlap:

• micro batch0: extend_seq_len = [1536], extend_prefix_len = [0]
• micro batch1: extend_seq_len = [1536], extend_prefix_len = [1536]

Compared to two-batch-overlap, the latency of group gemm, dispatch, and combine operations of the two micro batches is close in two-chunk-overlap, which is better for overlapping.

Can we add a small CI test to test_dp_attention.py?

I saw the test case of TBO in test_deepep_small. If TBO is ebabled, the two-chunk-overlap will be enabled in most cases. I think the test case of two-chunk-overlap can reuse the TBO, without the need for additional additions. And I have updated the parameter descriptions in server_arguments.md

@House-West @fzyzcjy @ch-wan I'd like to ask if you achieved performance improvements using TBO on two machines (1P1D)? Or did you achieve results without separating the PD (processing and data) on different machines? On my H20 system, performance decreased both with and without PD separation (1P1D). I've seen some publicly available experimental results online where TBO showed improvements, but they used many machines, such as 4P9D and 4P16D. I also tried your parameters, but the performance decreased significantly. Did I do something wrong? Or does TBO require a significant number of machines to achieve performance improvements? My rough analysis suggests that at least three machines are needed for the overhead and communication costs of TBO to be offset by the computational overlap, and perhaps five machines are needed to see any improvement. Looking forward to your reply！

@House-West @fzyzcjy @ch-wan I'd like to ask if you achieved performance improvements using TBO on two machines (1P1D)? Or did you achieve results without separating the PD (processing and data) on different machines? On my H20 system, performance decreased both with and without PD separation (1P1D). I've seen some publicly available experimental results online where TBO showed improvements, but they used many machines, such as 4P9D and 4P16D. I also tried your parameters, but the performance decreased significantly. Did I do something wrong? Or does TBO require a significant number of machines to achieve performance improvements? My rough analysis suggests that at least three machines are needed for the overhead and communication costs of TBO to be offset by the computational overlap, and perhaps five machines are needed to see any improvement. Looking forward to your reply！

@programmer-lxj I had tested on h800 previously , and there was no performance improvement with TBO on 1P1D (one machine for prefill, one machine for decode). Because intranode communication uses NVLink, which is relatively fast. In most cases, TBO can bring performance improvements when the communication time more than 30% of end-to-end time, such as 2P9D or 4P9D. For 1P1D , you can try SBO(single batch overlap).

@House-West Thank you very much! I will try SBO.

@House-West I tried using SBO, but I found that this parameter can only be added to the P node; adding it to the D node results in an error. I checked the code and found that the _combine_core function in sglang's deeprp.py passes the overlap_args dictionary parameter, but the Buff.low_latency_combine function in the deepep library doesn't have this parameter, which causes the error. Does this mean SBO can only be added to the P node and not the D node? Also, when using 1P1D, adding SBO only to the P node (and not the D node) resulted in a slight performance decrease, not an improvement. I wonder if there are other techniques I should be aware of.