SASC: Soft-Averaged Self-Consistency to Improve Chain-of-Thought Reasoning in Instruct-LLMs

Ruban Vishnu Pandian, Alliot Nagle, Hyeji Kim

Keywords: Chain-of-Thought Reasoning, Self-Consistency, Instruct-LLMs

TL;DR: We propose Soft-Averaged Self-Consistency, an inference time, probabilistic method for improving the reasoning performance of Chain-of-Thought Instruct-LLMs

Abstract: Improving the reasoning performance of LLMs during inference by combining scores of multiple output reasoning traces has emerged as a prominent technique, particularly for Chain-of-Thought Instruct-LLM models. Methods such as Self-Consistency and Self-Certainty assign simple yet effective metrics to the generated outputs which are then used to determine the final answer. These methods fit into the framework of designing a score vector over the set of generated answers, using weighted averaging of one-hot hard-decision vectors. In this work, we propose ***Soft-Averaged Self-Consistency***, a simple method in which the hard-decision vectors are replaced with the conditional probability vectors obtained from the LLM logits, forming our soft-decision vectors. We show that our theoretically motivated method provides improved accuracy when compared against other baselines, backed by extensive experiments on a variety of tasks and models.

Email Sharing: We authorize the sharing of all author emails with Program Chairs.

Data Release: We authorize the release of our submission and author names to the public in the event of acceptance.

Submission Number: 172