Multimodal Retrieval-Augmented Generation Question-Answering System
Keywords: Retrieval-Augmented Generation; Dataset Construction; Text-Image Retrieval; Visual Question-Answering System
TL;DR: My paper presents a Multimodal RAG Question-Answering System, which integrates text-to-image retrieval with multimodal models to enhance accuracy, efficiency, and retrieval in complex scenarios while simplifying document processing.
Abstract: Retrieval-Augmented Generation (RAG) combines the richness of external knowledge bases with the generative capabilities of large language models (LLMs) to provide users with more accurate and real-time responses. However, in the era of information explosion, the way information is presented is increasingly becoming multimodal. Users are no longer satisfied with the information provided by traditional text-based knowledge bases, making the construction of an efficient and accurate multimodal RAG question-answering system of significant theoretical and practical importance. To address these issues, this paper proposes an innovative RAG question-answering system: this approach pre-designs a rich dataset containing images, text, and question-answer pairs from external knowledge bases for subsequent model training, effectively improving the training quality of the model; it builds a cross-modal retrieval model from text to images, ensuring precise matching between document content and corresponding images, significantly reducing the complexity and processing time of locating relevant images within long texts. Furthermore, the retrieval model and the multimodal question-answering model are integrated to construct an efficient and accurate RAG question-answering system. Experimental results show that this system not only effectively simplifies the document formatting process and improves text-to-image retrieval accuracy but also exhibits comprehensive performance in handling multimodal data.
Primary Area: foundation or frontier models, including LLMs
Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
Submission Guidelines: I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.
Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.
No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.
Submission Number: 2880
Loading