Go to ACL ARR 2025 February homepageDocBench: A Benchmark for Evaluating LLM-based Document Reading Systems
Abstract: Recent advancements in proprietary large language models (LLMs), such as those from OpenAI and Anthropic, have led to the development of document reading systems capable of handling raw files with complex layouts, intricate formatting, lengthy content, and multi-modal information. However, the absence of a standardized benchmark hinders objective evaluation of these systems. To address this gap, we introduce DocBench, a benchmark designed to simulate real-world scenarios, where each raw file consists of a document paired with one or more questions. DocBench uniquely evaluates entire document reading systems and adopts a user-centric approach, allowing users to identify the system best suited to their needs.
Paper Type: Short
Research Area: Resources and Evaluation
Research Area Keywords: Document understanding, Agent
Contribution Types: Data resources
Languages Studied: English
Submission Number: 6374
Loading