BigMac: A Communication-Efficient Mixture-of-Experts Model Structure for Fast Training and Inference.

Zewen Jin, Shengnan Wang, Jiaan Zhu, Hongrui Zhan, Youhui Bai, Lin Zhang, Zhenyu Ming, Cheng Li