Mixture of Groups: Grouped Gating and Cross Mixing for Parameter-Efficient LLM Fine-Tuning

Yueqi Yang, Liu Yang, Xi-Le Zhao, Qilong Wang, Qinghua Hu

Keywords: large language model, parameter-efficient fine-tuning, low-rank adaptation

Abstract: Low-rank adaptation (LoRA) is a widely used parameter-efficient fine-tuning (PEFT) method for large language models. However, by learning independent adapters for each layer, LoRA and its variants ignore the inherent functional similarity between adjacent layers, limiting their potential to fully exploit the hierarchical representations across depth. To address this, we propose Mixture of Groups (MoG), a novel group-sharing framework that partitions layers into functional groups, shares low-rank adapters within each group, and employs adaptive gating and cross-mixing mechanisms to enable flexible fine-tuning. This approach leverages inter-layer similarity to capture both commonalities and unique characteristics across layers, achieving a more efficient and expressive subspace than LoRA. Moreover, MoG is designed as a plug-and-play framework that can be seamlessly integrated into other PEFT methods such as DoRA and PiSSA to boost their performance. Extensive experiments on multiple benchmarks demonstrate that MoG achieves overall superior performance compared with prior methods under comparable parameter budgets, highlighting its ability to combine efficiency with strong downstream effectiveness.

Primary Area: foundation or frontier models, including LLMs

Submission Number: 2593