tutel is slower than the naive p2p using 2DH for small scale

[DongyuXu77]

Hello, and thank you for providing this excellent codebase. I have a question regarding our experiment, where we observed that the 2DH inter-node communication is slower compared to the naive peer-to-peer approach, despite both methods transmitting the same amount of data. This observation contradicts our initial intuition, and I'm curious about the potential reasons behind it?

[ghostplant]

Did you run with large numbers of GPUs? In our experiment, 2DH gradually becomes faster than Linear A2A when the number of distributed A100s are at least over 256. Otherwise, 2DH is slower than Linear A2A in smaller scale.

[DongyuXu77]

Oh, that's may be the reason. In our experiment, the number of experts is 32 , which is less that 256

[ghostplant]

Yes, 2DH is faster to serve large scales only.