MergePRAG: Orthogonal Merging of Passage-experts for Multi-hop Parametric RAG

ICLR 2026 Conference Submission25133 Authors

Keywords: Multi-hop reasoning, Knowledge enhancement, Retrieval-augmented generation, Hypernetwork-based expert generation

Abstract:

Large language models (LLMs) can be enhanced with external knowledge through two dominant approaches: (1) $\textbf{retrieval-augmented generation (RAG)}$, which supplements LLMs with in-context retrieved passages, and (2) $\textbf{parametric knowledge adaptation (PKA)}$, which directly updates model parameters with new domain knowledge. Recently, parametric RAG (PRAG) has emerged as a promising framework, extending RAG by translating retrieved passages into parameter updates, thereby mitigating inefficiency and noise sensitivity inherent to RAG. However, existing PRAG methods remain limited to single-pass retrieval, falling short of the $\textbf{multi-hop RAG}$ setting that requires iterative retrieval and reasoning. We propose $\textbf{MergePRAG}$($\textit{Orthogonal Merging of Passage-experts for Multi-hop PRAG}$), a novel framework that sequentially integrates retrieved passages into LLM parameters through a continual merging mechanism, which is advanced by two key proposals: (1) $\textbf{orthogonal merging}$ using the Gram–Schmidt process to minimize conflicts between experts, and (2) $\textbf{critical-layer parameterization}$ to efficiently encode in-context passages. Experiments on multi-hop open-domain QA and reasoning-aware knowledge editing show that MergePRAG consistently outperforms both standard and state-of-the-art RAGs as well as existing parametric adaptation methods, achieving superior effectiveness and efficiency.

All datasets and code will be released at https://anonymous.4open.science/r/MhQA_hypernetwork-B31F.

Primary Area: applications to computer vision, audio, language, and other modalities

Submission Number: 25133