UniversalRAG: Retrieval-Augmented Generation over Corpora of Diverse Modalities and Granularities

ICLR 2026 Conference Submission16321 Authors

Keywords: RAG, Multimodal RAG, Data Granularity in RAG

TL;DR: UniversalRAG is a novel RAG framework that dynamically retrieves across multiple modalities and granularities by introducing a modality-aware routing mechanism.

Abstract: Retrieval-Augmented Generation (RAG) has shown substantial promise in improving factual accuracy by grounding model responses with external knowledge relevant to queries. However, most existing approaches are limited to a text-only corpus, and while recent efforts have extended RAG to other modalities such as images and videos, they typically operate over a single modality-specific corpus. In contrast, real-world queries vary widely in the type of knowledge they require, which a single type of knowledge source cannot address. To this end, we introduce UniversalRAG, designed to retrieve and integrate knowledge from heterogeneous sources with diverse modalities and granularities. Specifically, motivated by the observation that forcing all modalities into a unified representation space derived from a single aggregated corpus causes a modality gap, where the retrieval tends to favor items from the same modality as the query, we propose modality-aware routing that dynamically identifies the most appropriate modality-specific corpus and performs targeted retrieval within it. Also, beyond modality, we organize each modality into multiple granularity levels, enabling fine-tuned retrieval tailored to the complexity and scope of the query. We validate UniversalRAG on 8 benchmarks of multiple modalities, showing superiority over modality-specific and unified baselines.

Primary Area: applications to computer vision, audio, language, and other modalities

Submission Number: 16321