Universal Self-Consistency for Large Language Models

Xinyun Chen, Renat Aksitov, Uri Alon, Jie Ren, Kefan Xiao, Pengcheng Yin, Sushant Prakash, Charles Sutton, Xuezhi Wang, Denny Zhou

Track: long paper (up to 8 pages)

Keywords: Universal Self-Consistency, Large Language Model

TL;DR: We propose Universal Self-Consistency (USC), which leverages LLMs themselves to select the most consistent answer among multiple candidates.

Abstract: Self-consistency with chain-of-thought (CoT) prompting has demonstrated remarkable performance gain by utilizing multiple reasoning paths sampled from large language models (LLMs). However, self-consistency relies on heuristics to extract answers and aggregate multiple solutions, which is not applicable to solving tasks with free-form answers. In this work, we propose Universal Self-Consistency (USC), which leverages LLMs themselves to select the most consistent answer among multiple candidates. We evaluate USC on a variety of benchmarks, including mathematical reasoning, code generation, long-context summarization, and open-ended question answering. On open-ended generation tasks where the original self-consistency is not applicable, USC effectively leverages multiple samples and improves the performance. For math reasoning, USC matches the standard self-consistency performance without requiring the answer formats to be similar. Finally, without access to execution results, USC also performs on par with execution-based voting methods on code generation.

Submission Number: 41