Calc-X and Calcformers: Empowering Arithmetical Chain-of-Thought through Interaction with Symbolic Systems

Marek Kadlčík, Michal Štefánik, Ondrej Sotolar, Vlastimil Martinek

Submission Type: Regular Short Paper

Submission Track: Resources and Evaluation

Submission Track 2: Natural Language Generation

Keywords: arithmetic reasoning, multistep reasoning, dataset, generation

TL;DR: We create Calc-X datasets collection allowing to train Calcformer generative language models to offload the reasoning steps requiring arithmetic computation to precise symbolic systems such as a calculator.

Abstract:

Despite outstanding performance in many tasks, language models are notoriously inclined to make factual errors in tasks requiring arithmetic computation. We address this deficiency by creating Calc-X, a collection of datasets that demonstrates the appropriate use of a calculator in reasoning chains. Calc-X is suitable for teaching language models to offload computations to a symbolic system.
We survey and unify several existing chain-of-thought datasets into a proposed format, resulting in a standard collection of over 300,000 samples requiring arithmetic reasoning. Finally, we use the new Calc-X collection to train open-source calculator-using models and show that these models approximately double the accuracy of generating correct results compared to vanilla language model baselines.

Submission Number: 4189