Scientific logicality enriched methodology for LLM reasoning: A practice in physics

Zhaoxin Yu, Nan Xu, Kun Chen, Jiahao Zhao, Lei Wang, Wenji Mao

Keywords: scientific logicality, logicality assessment, LLM physics reasoning

TL;DR: We designed an algorithm to assess the logicality of LLMs' scientific reasoning and leveraged it to construct an SFT dataset for physics reasoning that exhibits high logicality.

Abstract: With the continuous advancement of reasoning abilities in Large Language Models (LLMs), their application to scientific reasoning tasks has gained significant research attention. Current research primarily emphasizes boosting LLMs' performances on scientific QA benchmarks by training on larger, more comprehensive datasets with extended reasoning chains. However, these approaches neglect the essence of scientific reasoning process -- logicality, which is the rational foundation to ensure the validity of reasoning steps leading to reliable conclusions. In this work, we make the first systematic investigation into the internal logicality underlying LLM scientific reasoning, and develop a scientific logicality enriched methodology, including a set of assessment criteria and data sampling methods for logicality-guided training, to improve the logical faithfulness as well as task performance. Further, we take physics, characterized by its diverse logical structures and formalisms, as an exemplar discipline to practise the above methodology. For data construction, we extract scientific problems from academic literature and sample a high-quality dataset exhibiting strong logicality. Experiments based on three different backbone LLMs reveal that: 1) the training data we constructed can effectively improve the scientific logicality in LLM reasoning; and 2) the enriched scientific logicality plays a critical role in solving scientific problems.

Primary Area: datasets and benchmarks

Submission Number: 4313