Ask in Any Modality: A Comprehensive Survey on Multimodal Retrieval-Augmented Generation

ACL ARR 2025 February Submission6634 Authors

Abstract: Large Language Models (LLMs) suffer from hallucinations and outdated knowledge due to their reliance on static training data. Retrieval-Augmented Generation (RAG) mitigates these issues by integrating external dynamic information for improved factual grounding. With advances in multimodal learning, Multimodal RAG extends this approach by incorporating multiple modalities such as text, images, audio, and video to enhance the generated outputs. However, cross-modal alignment and reasoning introduce challenges beyond those in unimodal RAG. This survey offers a structured and comprehensive analysis of Multimodal RAG systems, covering datasets, benchmarks, metrics, evaluation, methodologies, and innovations in retrieval, fusion, augmentation, and generation. We precisely review training strategies, robustness enhancements, and loss functions, while also exploring the diverse Multimodal RAG scenarios. In addition, we outline open challenges and future directions to guide research in this evolving field. This survey lays the foundation for developing more reliable AI systems that effectively leverage multimodal dynamic external knowledge bases. To support further research, all resources are publicly available.

Paper Type: Long

Research Area: Multimodality and Language Grounding to Vision, Robotics and Beyond

Research Area Keywords: retrieval-augmented models, retrieval-augmented generation, multimodality

Contribution Types: Surveys

Languages Studied: English, along with various other languages such as Chinese, supported by multilingual models.

Submission Number: 6634