FedMOPA: Federated Multi-Objective Preference Alignment for Large Language Models

Pengxin Guo, Baijiong Lin, Kunyang Song, Yuyin Zhou, Liangqiong Qu

Keywords: Federated Learning, Federated Multi-Objective Preference Alignment

Abstract: Aligning Large Language Models (LLMs) with diverse and often conflicting human preferences is a critical challenge, magnified in scenarios where preference data is distributed across multiple clients. In this paper, we propose **FedMOPA**, a novel framework that integrates federated learning with multi-objective optimization to align LLMs with diverse user preferences while preserving data privacy. Our core innovation is a unified, preference-conditioned model that dynamically adapts to varying trade-offs among client preferences at inference time, eliminating the need for retraining. To tackle the prohibitive communication costs of federated fine-tuning, we introduce **TriLoRA**, a conditional LoRA variant that efficiently injects preference information into the low-rank adaptation process. To mitigate the aggregation errors inherent in naively averaging TriLoRA parameters, we further design an alternating optimization strategy that ensures stable convergence and enhances model performance. We provide a theoretical analysis demonstrating the convergence of our method and its ability to achieve the Pareto front under certain conditions. Extensive evaluations on real-world datasets, such as safety alignment and helpful assistant tasks, confirm that FedMOPA effectively achieves superior preference alignment across multiple objectives. Our code is available at https://anonymous.4open.science/r/FedMOPA-10427.

Primary Area: transfer learning, meta learning, and lifelong learning

Submission Number: 10427