FedBridge: Bridging Domain Experts and Domain Knowledge via a Federated Learning Framework for Controlled Model Personalization

Chuantao Li, Zhigang Zhao, Xiaoming Wu, Jintao Li, Guangdong Zhang, Chunxiao Wang, Longxiang Gao

Keywords: Federated Learning, Domain LLM, RAG, PEFT

Abstract: The rapid development of large language models (LLMs) has highlighted a critical challenge in applying these models to domain-specific tasks while preserving data privacy. This study introduces FedBridge, a novel architecture that seamlessly bridges parameter-efficient fine-tuning (PEFT) and retrieval-augmented generation (RAG) within a federated learning framework, enabling the deep integration of domain expert and knowledge through these three pillars. Initially, we propose FF-LoRA (Federated Fusion Low-Rank Adaptation), a PEFT variant that fuses server-level global representations with client-specific parameters to mitigate client drift caused by heterogeneous local data. Following this, we design a dual-task strategy that constructs independent local case and global authoritative bases, enabling independent querying and targeted retriever optimization. Furthermore, we establish bidirectional consistency between the fine-tuned domain models and the retriever system: the domain model's output guides retriever to precisely identify latent relevant documents, while concurrently serving as a generator, thus improving coherence and domain fidelity of the retrieval-response pipeline. Experimental results demonstrate that the proposed architecture efficiently improves accuracy and robustness in both close-ended domain and open-ended domain tasks.

Primary Area: alignment, fairness, safety, privacy, and societal considerations

Submission Number: 8237