End-to-End Document Understanding via Chain-of-Reading

Jindi Guo, Chaozheng Huang, Haoyi Tao, Zhulin An, Guolin Ke, Xi Fang

Keywords: Document Intelligence, Multimodal Large Models, Chain-of-Thought, OCR

Abstract: Intelligent Document Analysis (IDA) is a formidable task owing to documents’ complex layouts, dense tables, charts, and mixed modalities. Conventional pipelines apply OCR before large language model reasoning but suffer from error propagation. End-to-end multimodal models avoid explicit pipelines yet struggle to scale to multi-page documents, where information dilution and evidence localization remain major bottlenecks. We propose Chain-of-Reading (CoR), an end-to-end framework that transforms traditional text-centric reading into a native multimodal paradigm. CoR directly consumes PDF pages as visual input, mimicking human eyes, and performs document-level question answering through a chain-of-thought process. It first localizes relevant evidence, then selectively applies OCR, and finally performs reasoning over the localized content. To further enhance comprehension of visual elements such as charts and scientific figures—which exacerbate information dilution and impede pinpointing evidence—we introduce Masked Auto-Regression (Mask-AR), a self-supervised method for multimodal grounding. CoR achieves a 14.3% improvement over the base model on the MMLongBench-Doc benchmark. We will release the CoR-Dataset and our fine-tuned model, Qwen2.5-VL-CoR.

Primary Area: foundation or frontier models, including LLMs

Submission Number: 10842