MergePRAG: Orthogonal Merging of Passage-experts for Multi-hop Parametric RAG

ICLR 2026 Conference Submission25133 Authors

Keywords: Multi-hop reasoning, Knowledge enhancement, Retrieval-augmented generation, Hypernetwork-based expert generation

Abstract:

Large language models (LLMs) can be enhanced with external knowledge through two dominant approaches: (1) **retrieval-augmented generation (RAG)**, which supplements LLMs with in-context retrieved passages, and (2) **parametric knowledge adaptation (PKA)**, which directly updates model parameters with new domain knowledge. Recently, parametric RAG (PRAG) has emerged as a promising framework, extending RAG by translating retrieved passages into parameter updates, thereby mitigating inefficiency and noise sensitivity inherent to RAG. However, existing PRAG methods remain limited to single-pass retrieval, falling short of the \textbf{multi-hop RAG} setting that requires iterative retrieval and reasoning.
We propose **MergePRAG**(*Orthogonal Merging of Passage-experts for Multi-hop PRAG*), a novel framework that sequentially integrates retrieved passages into LLM parameters through a continual merging mechanism, which is advanced by two key proposals: (1) **orthogonal merging** using the Gram–Schmidt process to minimize conflicts between "passage experts", and (2) **critical-layer parameterization** to efficiently encode in-context passages. Experiments on multi-hop open-domain QA and reasoning-aware knowledge editing show that MergePRAG consistently outperforms both standard and state-of-the-art RAGs as well as existing parametric adaptation methods, achieving superior effectiveness and efficiency. All datasets and code will be released at {https://anonymous.4open.science/r/MhQA_hypernetwork-B31F.

Primary Area: applications to computer vision, audio, language, and other modalities

Submission Number: 25133