RevOrder: A Novel Equation Format for Arithmetic Operations in Language Models
Abstract: This paper proposes to understand arithmetic operations in Language Models (LM) by framing them as digit-based reasoning challenges. We introduce a metric called the Count of Sequential Intermediate Digits (CSID), which measures the complexity of arithmetic equations by counting the missing steps in digit reasoning. Our empirical findings suggest that increasing the model size does little to improve the handling of equations with high CSID values.
We propose RevOrder, a method that incorporates techniques such as reversing the output order, step-by-step decomposition, and rollback mechanisms to maintain a low CSID, thereby enhancing the solvability of arithmetic equations in LMs. RevOrder also introduces a more compact reasoning process, which reduces the token requirements without affecting the CSID, significantly enhancing token efficiency.
Comprehensive testing shows that RevOrder achieves perfect accuracy in operations such as addition, subtraction, and multiplication, and substantially improves performance in division tasks, especially with large numbers where traditional models falter. Additionally, applying RevOrder to fine-tune the LLaMA2-7B model on the GSM8K math task led to a significant reduction in equation calculation errors by 46\% and increased the overall score from 41.6 to 44.4. The data and code can be found at https://anonymous.4open.science/r/RevOrder-D1E1.
Paper Type: Long
Research Area: Generation
Research Area Keywords: human evaluation, analysis, inference methods
Contribution Types: NLP engineering experiment, Approaches low compute settings-efficiency, Theory
Languages Studied: English, Math
Submission Number: 752
Loading