Semantic Self-Consistency: Enhancing Language Model Reasoning via Semantic Weighting

Tim Knappe, Ryan Luo Li, Ayush Chauhan, Kaylee Chhua, Kevin Zhu, Sean O'Brien

Keywords: Candidate responses, chain-of-thought prompting, reasoning path embeddings, fine-tuned BERT models, vector embeddings, weighting algorithms, aggregation, filtering, anomalous results, semantic relevance, decision-making enhancement.

TL;DR: We propose new methods building on top of self-consistency incorporating semantic rationale content.

Abstract: While large language models (LLMs) have rapidly improved their performance on a broad number of tasks, they still often fall short on reasoning tasks. As LLMs become more integrated in diverse real-world tasks, advancing their reasoning capabilities is crucial to their effectiveness in nuanced, complex problems. \citet{wang2023selfconsistency}’s \textit{self-consistency} framework reveals that sampling multiple rationales before taking a majority vote reliably improves model performance across various closed-answer reasoning tasks. Standard methods based on this framework aggregate the final decisions of these rationales but fail to utilize the semantic information detailed in the step-by-step reasoning paths. Our work introduces \textit{semantic self-consistency}, enhancing this approach by incorporating and analyzing both the reasoning paths of these rationales in addition to their final decisions before taking a majority vote. These methods not only improve the reliability of reasoning paths but also cause more robust performance on complex reasoning tasks.

Concurrent Submissions: Under submission at COLING

Submission Number: 111