MOP: Efficient Low-rank PHM Mixture of Experts for Prefix-based Multi-scenario Dialogue Summarizaton

Anonymous

Abstract: As large-scale pre-training models (PLMs) expand, efficient fine-tuning becomes crucial for rapid adaptation and deployment. We propose MOP, a low-rank Mixture of Experts (MOE) network for Prompt reparameterization in multi-scenario summarization based on prefix-tuning. MOP assigns specific experts for summarization in each particular scenario and incorporates an efficient knowledge decoupling mechanism. Specifically, Expert weight matrices are learned as a sum of Kronecker products of shared global and specific local weights, capturing general and task-specific knowledge. We further decompose global weights into low-rank layer-share (LoRL) and expert-share (LoRE) weights, enhancing flexibility and generality. By updating only the MOP, our method outperforms strong baselines across all scenarios on the MultiSum benchmark, using just 2.93% of a pretrained model's parameters, demonstrating MOP's effectiveness in improving multi- scenarios learning performance with fewer parameters.

Paper Type: long

Research Area: Generation

Contribution Types: Model analysis & interpretability, Approaches to low-resource settings, Data analysis, Surveys

Languages Studied: English, Chinese