It Ain't Over: A Multi-aspect Diverse Math Word Problem Dataset

Jiwoo Kim, Youngbin Kim, Ilwoong Baek, JinYeong Bak, Jongwuk Lee

Submission Type: Regular Long Paper

Submission Track: Resources and Evaluation

Submission Track 2: Multilinguality and Linguistic Diversity

Keywords: Math Word Problem, Arithmetic Reasonong, Natural Language Processing, Dataset, Large Language Model

Abstract: The math word problem (MWP) is a complex task that requires natural language understanding and logical reasoning to extract key knowledge from natural language narratives. Previous studies have provided various MWP datasets but lack diversity in problem types, lexical usage patterns, languages, and annotations for intermediate solutions. To address these limitations, we introduce a new MWP dataset, named DMath (Diverse Math Word Problems), offering a wide range of diversity in problem types, lexical usage patterns, languages, and intermediate solutions. The problems are available in English and Korean and include an expression tree and Python code as intermediate solutions. Through extensive experiments, we demonstrate that the DMath dataset provides a new opportunity to evaluate the capability of large language models, i.e., GPT-4 only achieves about 75\% accuracy on the DMath dataset.

Submission Number: 4935