Router Upcycling: Leveraging Mixture-of-Routers in Mixture-of-Experts Upcycling.

Junfeng Ran, Guangxiang Zhao, Yuhan Wu 0001, Dawei Zhu, Longyun Wu, Yikai Zhao 0001, Tong Yang 0003, Lin Sun 0010, Xiangzheng Zhang, Sujian Li