Too long; didn't solve

Lucía Cabrera, Isaac Saxton-Knight

Keywords: large language models, mathematical reasoning, benchmark evaluation, prompt length, solution length, structural features, model failure, cross-model disagreement, adversarial datasets, mixed-effects modeling

TL;DR: Longer math problems—both in their statements and solutions—make LLMs fail more often, with longer prompts also hinting at reduced separation between models.

Abstract: Mathematical benchmarks consisting of a range of mathematics problems are widely used to evaluate the reasoning abilities of large language models, yet little is known about how their structural properties influence model behaviour. In this work, we investigate two structural length variables, prompt length and solution length, and analyse how they relate to model performance on a newly constructed adversarial dataset of expert-authored mathematics problems. We find that both prompt and solution lengths correlate positively with increased model failure across models. We also include a secondary, exploratory analysis of cross-model disagreement. Under a difficulty-adjusted normalised analysis, both variables retain weak negative associations with realised model separation, slightly stronger for prompt length. Overall, our main robust finding is that structural length is linked to empirical difficulty in this dataset.

Email Sharing: We authorize the sharing of all author emails with Program Chairs.

Data Release: We authorize the release of our submission and author names to the public in the event of acceptance.

Submission Type: Research Paper

Archival Status: Archival

Submission Number: 46