Dynamic Voting for Efficient Reasoning in Large Language Models

Mingfeng Xue, Dayiheng Liu, Wenqiang Lei, Xingzhang Ren, Baosong Yang, Jun Xie, Yidan Zhang, Dezhong Peng, Jiancheng Lv

Submission Type: Regular Long Paper

Submission Track: Efficient Methods for NLP

Submission Track 2: Theme Track: Large Language Models and the Future of NLP

Keywords: large language models, multi-path voting, computational resource, early exiting

TL;DR: A multi-path voting technique that effectively reduces the number of reasoning paths while preserving accuracies by applying early exiting for problems that large language models can confidently solve.

Abstract:

Multi-path voting methods like Self-consistency have been used to mitigate reasoning errors in large language models caused by factual errors and illusion generation.
However, these methods require excessive computing resources as they generate numerous reasoning paths for each problem.
And our experiments show that on the arithmetic reasoning task, SVAMP, half of the problems fail to obtain noticeable accuracy gains when voting with more than three paths.
In this paper, we propose a novel multi-path voting technique called Dynamic Voting, which effectively reduces the number of reasoning paths during multi-path voting while preserving accuracies by applying early exiting for problems that large language models can confidently solve.
Experimental evaluations on arithmetic, commonsense, and symbolic reasoning tasks under few-shot and zero-shot settings demonstrate that Dynamic Voting achieves comparable accuracies employing significantly fewer reasoning paths.
Notably, one of our Dynamic Voting strategies outperforms Self-consistency using only 24.7\% of the number of paths on the LetterConcat task in the few-shot setting.
Furthermore, Dynamic Voting showcases strong robustness in threshold selection.
It also demonstrates excellent generalizability when combined with other voting techniques, different models, and diverse prompts.

Submission Number: 2770