Consistent Solutions for Optimizing Search Space of Beam Search

Yehui Xu, Sihui Li, Chujun Pu, Jin Wang, Xiaobing Zhou

Abstract: Research on math word problems has made significant advancements due to the emergence of language models. Large language models have excelled in a variety of reasoning tasks. Still, due to the demand for low costs, research on the upper bound of small language models in reasoning tasks and the limitation of the knowledge they can accommodate has drawn attention. In line with previous work on math word problems, we discover that models that only learned a single solution lacked reasoning ability during the decoding process, further exacerbating the error accumulation caused by exposure bias that will fail generalization. To tackle this problem, we suggest using the commutative property to generate a consistent solution set for each data in the training set. Then, we use it as additional training data to optimize the search space in beam search. On this foundation, we will go into great detail about how consistent solutions training affects the work process of beam search. In addition, we found significant differences between models trained using consistent solutions and those trained without consistent solutions, so the model ensemble technique is applied to improve model performance. In the NLPCC-2023 shared task 3, our model ultimately ranks fourth with an accuracy of 23.66%.