Self-Training Language Models in Arithmetic Reasoning
Keywords: large language models, dataset, arithmetical reasoning, math word problems, tool-using, self-training, supervised learning, preference optimization methods
TL;DR: We create Calc-X, a dataset for training calculator-using language models, train Calcformer models for solving math problems, and explore preference-optimization methods in self-training to improve reasoning.
Abstract: Recent works show the impressive effectiveness of an agent framework in solving problems with language models. In this work, we apply two key features from the framework, interaction with tools and goal-oriented training, to improve models' arithmetical reasoning.
First, we curate and transform existing datasets to create Calc-X, a standardized collection with over 300,000 problems with step-by-step solutions. We use Calc-X to train models we call Calcformers that interact with a calculator during inference. Calcformers achieve twice the accuracy of standard baselines.
Finally, we optimize Calcformers via self-training using preference optimization and supervised loss by checking the model's predicted results. We find that self-training can achieve substantial improvements on out-of-domain problems and that traditional supervised loss is a strong baseline for preference optimization. Our results show that preference optimization converges faster and isn't prone to forgetting pre-trained abilities.
Submission Number: 120
Loading