2Columns1Row: A Russian Benchmark for Textual and Multimodal Table Understanding and Reasoning

ACL ARR 2025 May Submission6094 Authors

Abstract:

Table understanding is a crucial task in document processing and is commonly encountered in practical applications. We introduce 2Columns1Row, the first open-source benchmark for the table question answering task in Russian. This benchmark evaluates the ability of models to reason about the relationships between rows and columns in tables, employing both textual and multimodal inputs.
2Columns1Row consists of six datasets, 28,680 tables, designed datasets that vary in the complexity of the text within the table contents and the consistency of the values in the cells. We evaluate the models using text-only and multimodal approaches and analyze their performance. Through extensive evaluation, we demonstrate the limitations of current multimodal models on this task and prove the feasibility of a dynamic text-based system utilizing our benchmark. Our results highlight significant opportunities for advancing table understanding and reasoning, providing a solid foundation for future research in this domain.

Paper Type: Long

Research Area: Resources and Evaluation

Research Area Keywords: table QA, benchmarking, automatic creation and evaluation of language resources, NLP datasets, vision question answering

Contribution Types: Model analysis & interpretability, Data resources

Languages Studied: Russian

Submission Number: 6094