UniDoc-Bench: A Unified Benchmark for Document-Centric Multimodal RAG

ACL ARR 2026 January Submission2396 Authors

Keywords: multimodal, evaluation, vqa, RAG, LLMs

Abstract:

Multimodal retrieval-augmented Generation (MM-RAG) is a key approach for applying large language models (LLMs) and agents to real-world knowledge bases, yet current evaluations are fragmented—focusing on either text or images in isolation, or simplified multimodal setup, failing to capture document-centric multimodal use cases.
In this paper, we introduce UniDoc-Bench, the first large-scale, realistic benchmark for MM-RAG built from $70$k real-world PDF pages across $8$ domains.
Our pipeline extracts and links evidence from text, tables, and
figures, then generates $1,600$ multimodal QA pairs spanning factual retrieval, comparison, summarization, and logical reasoning queries.
To ensure reliability, all of QA pairs
are validated by multiple human annotators and expert adjudication.
UniDoc-Bench supports apples-to-apples comparison across four paradigms --- 1) text-only, 2) image-only, 3) \emph{multimodal} text–image fusion and 4) \emph{multimodal} joint retrieval --- under a unified protocol with standardized candidate pools, prompts, and evaluation metrics.
UniDoc-Bench can also be used to evaluate Visual Question Answering (VQA) tasks.
Our experiments show that multimodal text–image fusion RAG systems consistently outperform both unimodal and jointly multimodal embedding–based retrieval, indicating that neither text nor images alone are sufficient and that current multimodal embeddings remain inadequate. Beyond benchmarking, our analysis reveals when and how visual context complements textual evidence, uncovers systematic failure modes, and offers actionable guidance for developing more robust MM-RAG pipelines.

Paper Type: Long

Research Area: Resources and Evaluation

Research Area Keywords: Language Modeling, Multimodality and Language Grounding to Vision, Robotics and Beyond

Contribution Types: Publicly available software and/or pre-trained models, Data resources

Languages Studied: English

Submission Number: 2396