Drop-Upcycling: Training Sparse Mixture of Experts with Partial Re-initialization

Open Webpage

Taishi Nakamura, Takuya Akiba, Kazuki Fujii, Yusuke Oda, Rio Yokota, Jun Suzuki

Published: 01 Jan 2025, Last Modified: 30 Jun 2025ICLR 2025EveryoneRevisionsBibTeXCC BY-SA 4.0