% Title: Large Language Models are Better Reasoners with Self-Verification

% ArXiv ID: 2212.09561



\maketitle
\begin{abstract}
Recently, with the chain of thought (CoT) prompting, large language models (LLMs), e.g., GPT-3, have shown strong reasoning ability in several natural language processing tasks such as arithmetic, commonsense, and logical reasoning. However, LLMs with CoT require multi-step prompting and multi-token prediction, which is highly sensitive to individual mistakes and vulnerable to error accumulation. The above issues make the LLMs need the ability to verify the answers. In fact, after inferring conclusions in some thinking decision tasks, people often check them by re-verifying steps to avoid some mistakes. In this paper, we propose and prove that LLMs also have similar self-verification abilities. We take the conclusion obtained by CoT as one of the conditions for solving the original problem. By performing a backward verification of the answers that LLM deduced for itself, we can obtain interpretable answer validation scores to select the candidate answer with the highest score. Experimental results demonstrate that the proposed method can improve the reasoning performance on various arithmetic, commonsense, and logical reasoning datasets. Our code is publicly available at: \url{https://github.com/WENGSYX/Self-Verification}.
\end{abstract}

\section{Introduction}
The ability of reasoning in the process of thinking and decision-making is an essential aspect of human intelligence. Recently, chain of thought (CoT) prompting \citep{wei2022chain} has been a good way to solve the arithmetic, commonsense, and logical reasoning tasks with large language models (LLMs), which help the LLMs simulating the human thinking process when solving complex natural language processing (NLP) tasks. CoT guides LLMs to generate a series of intermediate reasoning steps to address complex problems rather than just predict a final answer. This approach has been shown the advance performances on several challenging NLP tasks, even when using only a few or no training samples \citep{AmanMadaan2022LanguageMO,AbulhairSaparov2022LanguageMA,YaoFu2022ComplexityBasedPF,MIR-2022-09-289}.

\begin{figure*}[t]
\begin{center}
	\includegraphics[scale=0.53]{motivation2.pdf}
\end{center}
\caption{The answer of a question can be verified by masking and predicting the conditions of the original contexts. To mimic the self-verification ability of human, we predict the accuracy of $f_\mathcal{C}$ by predicting the original conditions $f_1$ or $f_2$ is right or not based on this conclusion.}
\label{idea}
\end{figure*}


Although CoT can enable LLMs to solve complex reasoning tasks, it is highly sensitive to individual mistakes and vulnerable to error accumulation~\citep{shen2021generate}. If a tiny mistake occurs, it can change the meaning deviations of the whole statement \cite{https://doi.org/10.48550/arxiv.2204.09269}, leading to incorrect answers \citep{cobbe2021training}. That is especially problematic in using CoT for addressing multi-step precise reasoning (such as mathematical calculation). Due to the lack of the error correction mechanism, it is difficult for the LLMs to obtain correct results from the possible errors in multiple steps reasoning. Detecting and mitigating errors is essential to improve reasoning capabilities.


Previous methods resolve the above issue by training another verifier to evaluate the correctness of the model's output \citep{shen2021generate,li2022advance}. However, there are some drawbacks in these work. On the one hand, training a verifier requires a lot of human annotations and additional fine-tuned models, which limits its widespread use in other tasks and domains. On the other hand, the verifier fine-tuned by a language model is not easily explainable, making it difficult to assess the model's reliability based on its output scores. Therefore, \textbf{the challenge of obtaining a better reasoner based on the LLMs is to get a verifier that can avoid manual annotation and additional training}, so that it can be better extended and migrated to other fields and tasks.

To address this challenge and overcome the limitations of training verifiers, we propose utilizing LLMs as reasoners with self-verification for selecting better prediction results. In numerous decision-making tasks, humans often perform self-verification of inferred conclusions to mitigate mistakes \cite{poole2010artificial}. In this paper, we propose and demonstrate that LLMs possess a similar self-verification ability, the better reasoning with CoT is carried out in the following two steps, \textbf{Forward Reasoning} and \textbf{Backward Verification}. Specifically, in Forward Reasoning, LLM reasoners generate candidate answers using CoT, and the question and candidate answers form different conclusions to be verified. And in Backward Verification, We mask the original condition and predict its result using another CoT. We rank candidate conclusions based on a verification score, which is calculated by assessing the consistency between the predicted and original condition values. For example, as shown in Figure \ref{idea}, by taking $f_2$ and $f_\mathcal{C}$ as conditions to predict the value of condition attribute in $\hat{f_1}$, the correctness of $f_\mathcal{C}$ can be evaluated by comparing the consistency of values of the predicted $\hat{f_1}$ and the original $f_1$ in verification. 



Our method employs LLMs for self-verification with only a few prompts, eliminating the need for fine-tuning or gradient updating. This approach enables automatic verification of multiple candidate answers and corresponding conclusions, mitigating deviations from the correct thought chain in the original CoT. Our verification score arises from evaluating each step during the backward verification phase, rather than from the direct output of a neural network model \citep{cobbe2021training,li2022advance}, enhancing the explainability of prediction outcomes and solution processes \citep{li2021more,Yu2023TowardsBC,zhu2023learning}. We conducted experiments on various open-source datasets for mathematical reasoning, common sense, and logical reasoning tasks, achieving results beyond the baseline (e.g., $60.8 \rightarrow 65.1$ on GSM8K, $91.01 \rightarrow 93.40$ on SingleEq). In addition, we also attempt to combine our method with some approaches to improving forward reasoning, such as self-consistency \citep{wang2023selfconsistency}  and Least-to-Most \citep{zhou2023leasttomost}. The experimental results show that our method also improves upon these forward reasoning approaches.

Our contributions are summarized as follows:

\begin{enumerate}
    \item We propose and prove that large language models (LLMs) can self-verify their prediction results. The proposed method can provide interpretable verification scores without the need for train additional verifiers. 
\item We have conducted extensive of experiments with multiple LLMs, and the experimental results on multiple mathematical, commonsense, and logical reasoning datasets achieve a significant improvement compared to the baseline. 
    \item We introduced True-False Item Verification for General Tasks in the backward verification stage and proposed Condition Mask Verification based on the characteristics of Arithmetic Tasks. Our method can be applied to a wide range of reasoning datasets, potentially paving the way for self-validation to become a new paradigm following pre-training and prompt learning, thus motivating further exploration of the capabilities of LLMs.
\end{enumerate}





\section{Related Work}

\textbf{Language Model Reasoning}. It has been extensively studied in order to evaluate the various reasoning abilities of language models \citep{SimranArora2022AskMA,AmanMadaan2022LanguageMO,MIR-2022-03-068}, including arithmetic reasoning \citep{RikKoncelKedziorski2015ParsingAW,SubhroRoy2016SolvingGA,ArkilPatel2021AreNM,cobbe2021training}, commonsense reasoning \citep{AlonTalmor2018CommonsenseQAAQ,ChandraBhagavatula2019AbductiveCR,MorGeva2021DidAU,Zhu2022ReasonChainQATC}, and logical reasoning \citep{JianLiu2020LogiQAAC,WeihaoYu2020ReClorAR}. To solve these reasoning tasks, researchers have utilized pre-trained language reasoning models \citep{AkariAsai2020LogicGuidedDA,XiangDeng2021ReasonBERTPT,Xia2022MedConQAMC} or fine-tuned general LLMs \citep{cobbe2021training}. Early work attempted to solve complex reasoning tasks using Seq2Seq models \citep{LeiWang2018TranslatingAM,JieruiLi2019ModelingII}. Later, specialized encoder-decoder architectures were designed to improve reasoning performance \citep{YibinShen2020SolvingMW,Zhu2022AKS}. More recent work has suggested to adopt pre-training tasks to improve arithmetic reasoning ability \citep{OriYoran2021TurningTG,wang-etal-2022-logic}. However, these methods require a significant amount of human annotation. In this paper, we proposed to obtain answers automatically and verify them in multiple reasoning tasks. 

\textbf{In-context Learning}. Large language models such as GPT-3 exhibit impressive few-shot learning ability \citep{JinghuiLu2022WhatMP,ShuofeiQiao2022ReasoningWL}, and closely approximate the predictors computed by gradient descent \citep{akyurek2022learning}. It requires only filling a few exemplars into context as prompts and without the need for finetuning on a dataset of training examples \citep{Wang2022TowardsUC,Weng2023LargeLM}. However, this approach struggles with tasks requiring complex reasoning \citep{DBLP:journals/corr/abs-2112-11446}, which drives researchers to explore other prompting strategies. CoT \citep{wei2022chain} is a chained reasoning approach that inserts a multi-step reasoning path before generating the final answer. \citet{XuezhiWangSelfConsistencyIC} proposed a self-consistency decoding strategy to vote on the reasoning path, and \citet{kojima2022large} demonstrated that LLMs could as zero-shot reasoners through the prompt ``Let's think step-by-step''. These methods focus on constructing the CoT but ignore the high sensitivity of LLMs to individual mistakes in generating these chains, so some of these conclusions by CoT may be unreliable \citep{Dhuliawala2023ChainofVerificationRH,Chu2023ASO,weng2023mastering}. In this paper, we proved that LLMs can self-verify their conclusions.

\textbf{Answer Verification}. It is a common method for evaluating and reordering candidate answers with a trained language understanding model. \citet{kushman-etal-2014-learning} train a classifier to select the best answer from candidate answers, while \citet{SubhroRoy2016SolvingGA} train a global scoring model to guide the search process for better answers. \citet{shen2021generate} proposed the joint training of answer generation and rank with language model. \citet{cobbe2021training} and \citet{Lightman2023LetsVS} fine-tunes language model as a verifier, which calculates token-level and solution-level verification scores for a predicate result. However, the above method all need additional annotations. In our work, we do not require training examples and can provide an explainable verification score.

\begin{figure*}[t]
\begin{center}
	\includegraphics[scale=0.46]{methodv8.pdf}
\end{center}
\caption{Example of self-verification. In the step one, LLM generates candidate answers and forms different conclusions. Then, in the step two, LLM verifies these conclusions in turn and computes the verification score.}
\label{method}
\end{figure*}

\section{The Proposed Method}
The proposed method can be used to verify prediction results. As shown in Figure \ref{method}, the process mainly consists of two steps. The first step, forward reasoning, is similar to the normal CoT, except that multiple candidate answers are generated through sampling decoding. In the second step, we calculate the verification scores for each candidate's answer by the self-verification method, and the answer with the highest score is selected as the final answer.


\subsection{Forward Reasoning}
In forward reasoning, the LLM reasoners generate candidate answers with the chain of thought prompting. We augment the input with several CoT prompts similar to the original query and then send it to the LLM. The LLM then performs sampling decoding to generate multiple candidates for verification.

As shown in Figure \ref{method}, for a reasoning task, the large language model $\mathcal{LLM}$ is given a question $\mathcal{X}$ which is accompanied by a chain of thought set $\mathrm{C}$. In few-shot setting, the whole prompt also contains other question-CoT prompt-answer tuples. The input $\mathcal{X}$ can be further subdivided into $\mathcal{X} = \{f_1, f_2,\dots, f_R, q\}$, where each $f_i$ is a condition (fact), and $q$ is a question, both represented as natural language clause or sub-sentences. 

Specifically, in order to generate step-by-step solutions with CoT, we followed \citet{wei2022chain} and designed CoT prompt set $\mathrm{C}$ for the reasoning dataset (e.g., the GSM8K dataset), which contains $n$ samples, each sample has the question $\mathcal{\dot{X}}$, chain of thout $\dot{t}$, and the answer $\dot{y}$. These samples are used as the input of test-time. Each example in $\mathrm{C}$ is concatenated as a prompt:
\setlength\arraycolsep{1pt}\begin{eqnarray*}
\mathrm{C} = (\mathcal{\dot{X}}_0,\mathbf{\dot{t}}_0,\mathbf{\dot{y}}_0); (\mathcal{\dot{X}}_1,\mathbf{\dot{t}}_1,\mathbf{\dot{y}}_1);\dots;(\mathcal{\dot{X}}_n,\mathbf{\dot{t}}_n,\mathbf{\dot{y}}_n)
\end{eqnarray*}

Therefore, $\mathcal{LLM}$ is required to follow the prompt of $\mathrm{C}$ to generate the chain of thought $\mathbf{t_{CoT}}$ before generating the final answer $\mathbf{y}$:

\setlength\arraycolsep{1pt}\begin{eqnarray*}
\mathcal{P}(\mathbf{y}|\mathrm{C},\mathcal{X}) = \mathcal{P}(\mathbf{t_{CoT}}|\mathrm{C},\mathcal{X}) \times \mathcal{P})(\mathbf{y}|\mathrm{C},\mathcal{X},\mathbf{ t_{CoT}})
\end{eqnarray*}


To ensure the diversity of different answers, we adapt sampling decoding \citep{radford2019language} to generate multiple $\mathbf{y}$ for $K$ times. Specifically, sampling decoding is a random decoding method, which can select the next word by sampling from a probability distribution over the possible words at each step. Multiple candidate answers can be obtained when repeatedly using sampling decoding. For example, we generate \textit{``18''} and \textit{``2''} as candidate answers in the example of Figure \ref{method}. 


\subsection{Backward Verification} % 第一步，把问题和答案形成conclusion；第二步，轮流选择原来条件中的事实，通过正则提取属性值，形成新的待预测对象；第三步，推理并比较。

Step 1 may generate multiple different answers, this step is used to verify and select the best answer. Backward verification involves several sub-steps. First, the original question with each candidate's answer is rewritten as a conclusion and then supplemented as a new condition (incarnadine color in Figure \ref{method}). Then, we considered two methods to construct new questions. In the general QA task, the True-False Item Verification is given based on all the conditions, asking the LLM whether these conditions are mutually satisfied, it has a broad applicability. In Arithmetic reasoning tasks, as the definite condition masks can indicate the reasoning direction of the language model, we propose the Condition Mask Verification method to design questions for the verification stage. Finally, we perform multiple experiments to compute the verification score by comparing the consistency between the predicted condition value and the original masked condition value, and select the candidate answer with the highest score as the final answer.

\subsubsection{Rewritten Candidate Conclusion}


Besides, we rewrite the original question with the candidate's answer as a conclusion and then supplement it as a new condition in the backward verification step. Specifically, we use the instruction prompt \textit{``Please change the questions and answers into complete declarative sentences [$q$] The answer is [$\mathbf{y}$]''} to change $q$ and $\mathbf{y}$ into new declarative sentence $f_\mathcal{Y}$ by $\mathcal{LLM}$. As shown in Figure \ref{method}, we can rewrite the question and conclusion as \textit{``Jackie has 18 apples more than Adam''}.


\subsubsection{Condition Masking}
For question generation, the diversity of the problems makes it difficult to balance the need for coherence and fact consistency between questions and answers in practical operation \cite{sun-etal-2018-answer, TianboJi2022QAScoreA}. To tackle this issue, we included clear questions asking the language model to accurately predict.

\textbf{True-False Item Verification (TFV).} This approach can be applied to a wide range of reasoning QA tasks. We directly add "Do it is correct (True or False)?" after all the conditions, requiring the LLM to self-evaluate the correctness of these conditions.


\textbf{Condition Mask Verification (CMV).} Further, we use regular expressions to filter out specific conditions, such as numbers, and then mask them in turn. If we do not mask all conditions but randomly select a condition, unnecessary conditions may be masked, which will significantly impact the verification answer. For example, ``\textit{Dana worked 9 hours on Friday, 10 hours on Saturday, and 3 hours on Sunday. She earns \$13 per hour. How much money did Dana earn in weekend?}'', since condition 1 (9 hours) does not affect the conclusion, it is difficult to predict it correctly. We replace all occurrences of $f$ in the original $\mathcal{X}$ with \textit{``X''} in turn, and ask $\mathcal{LLM}$ to re-predict it. Then we rewrite the question. For example, we might find a value in $f_1$ and replace it with \textit{``X''}. We can then add \textit{``What is the answer of `X'?''} to the end of the new question, effectively turning it into an equation. This technique helps to guide the language model towards the correct answer.


\subsubsection{Verification Score Calculation}
This backward verification chain of thought is similar to solving an equation. We design a chain of thought prompt, like forward reasoning, to guide LLM in generating a solving process. We input the newly constructed sentences into $\mathcal{LLM}$. For TFV, we can directly count the number of answers that are True as the score, and for CMV, we will match its final result with the masked condition.



Due to the limited performance of LLM itself, if the condition is verified only once in the backward verification step, it is easy to have the same score, resulting in a lack of differentiation. To address this, we repeat the sampling decoding process $P$ times, so that the verification score can more accurately reflect the model's confidence for a given conclusion \citep{erd1970new}.

The verification score is calculated as follows:
\setlength\arraycolsep{2pt}

\begin{eqnarray*}
\resizebox{0.45\textwidth}{!}{%
$\text{Score}_\mathbf{y} = 
\left\{\begin{array}{l}

\sum_{p=1}^P(\sum_{r=1}^R1_{(\mathcal{LLM}_p(\mathcal{X}-{f_r}+{f_\mathcal{Y}}) = f_r)}) \quad \text{TFV} \\


\sum_{p=1}^P(1_{(\mathcal{LLM}_p(\mathcal{X}+{f_\mathcal{Y}}))}) \quad \text{CMV}
\end{array}\right.$}
\end{eqnarray*}

Where $1_{(\bullet)}$ is an indicator function. 

Finally, we select the one with the highest verification score from the K candidate answers generated as a result.
\setlength\arraycolsep{2pt}\begin{eqnarray*}
\mathbf{Output} = \mathop{\mathrm{argmax}}_{k \in [0,K]} (\mathbf{Score}_k)
\end{eqnarray*}
\par For example for CMV, in Figure \ref{method}.3)Verification, we match the results generated by the self-verification of LLM with the masked conditions. There is one \textit{``10''} in the conclusion of $A_1$, so the verification score is $1$. There are four correct results in $A_2$, so the verification score is $4$, and we finally choose $A_2$, which has the highest verification score, as the final conclusion.



\begin{table*}[h]
\centering

\renewcommand\arraystretch{1.5}
\resizebox{\textwidth}{!}{%
\begin{tabular}{cc|llllll|ll}

 \bottomrule \bottomrule
 &&\multicolumn{6}{|c|}{Arithmetic Tasks}&\multicolumn{2}{|c}{General Tasks} \\
\multicolumn{2}{c|}{Method} &GSM8K&SingleEq&AddSub&MultiArith&AQUA-RAT&SVAMP&CSQA&DU  \\
  \bottomrule
  \multicolumn{2}{c|}{Previous SOTA (Fine-tune)} &$35^a$/$57^b$&$32.5^c$&$94.9^d$&$60.5^e$&$37.9^f$&$57.4^g$&$91.2^h$& - \\
 \multicolumn{2}{c|}{9–12 year olds}   &$60^i$&-&-&-&-&-&-&- \\ 
 \multicolumn{2}{c|}{GPT-3 Standard}&19.7&86.8&90.9&44.0&29.5&69.9&82.3&49.0 \\
  \bottomrule
  \multirow{2}{3.1cm}{\centering GPT-3 (175B)\\{\tt{}code-davinci-001}}&CoT&13.84&60.20&58.55&45.85&18.90&38.42&46.75&38.72   \\
  &CoT+\textbf{Self-Verification}&$\textbf{13.92}_{\textcolor[RGB]{34,139,34}{(+0.08)}}$&$\textbf{60.61}_{\textcolor[RGB]{34,139,34}{(+0.41)}}$&$\textbf{59.07}_{\textcolor[RGB]{34,139,34}{(+0.52)}}$&$\textbf{46.19}_{\textcolor[RGB]{34,139,34}{(+0.34)}}$&$\textbf{27.04}_{\textcolor[RGB]{34,139,34}{(+8.14)}}$&$\textbf{38.96}_{\textcolor[RGB]{34,139,34}{(+0.54)}}$&$\textbf{47.68}_{\textcolor[RGB]{34,139,34}{(+0.93)}}$&$\textbf{39.03}_{\textcolor[RGB]{34,139,34}{(+0.31)}}$

   \\




   
  \bottomrule
  \multirow{2}{3.1cm}{\centering Instruct-GPT (175B)\\{\tt{}code-davinci-002}}&CoT&60.81&91.01&82.78&96.13&45.30&75.87&77.42&65.43	   \\
  &CoT+\textbf{Self-Verification}&$\textbf{65.14}_{\textcolor[RGB]{34,139,34}{(+4.33)}}$&$\textbf{93.40}_{\textcolor[RGB]{34,139,34}{(+2.39)}}$&$\textbf{86.33}_{\textcolor[RGB]{34,139,34}{(+3.55)}}$&$\textbf{99.15}_{\textcolor[RGB]{34,139,34}{(+3.02)}}$&$\textbf{47.95}_{\textcolor[RGB]{34,139,34}{(+2.65)}}$&$\textbf{76.99}_{\textcolor[RGB]{34,139,34}{(+1.12)}}$&$\textbf{77.83}_{\textcolor[RGB]{34,139,34}{(+0.41)}}$&$\textbf{66.57}_{\textcolor[RGB]{34,139,34}{(+1.14)}}$   \\

       \bottomrule
  \bottomrule 
\multicolumn{10}{c}{Self-Consistency Decoding \cite{XuezhiWangSelfConsistencyIC} For Forward Reasoning}\\
  \bottomrule
       \bottomrule
\multirow{2}{3.1cm}{\centering GPT-3 (175B)\\{\tt{}code-davinci-001}}&SC&23.40&70.25&68.65&79.82&25.60&54.58&54.92&49.26  \\
  &SC+ \textbf{Self-Verification}&$\textbf{23.59}_{\textcolor[RGB]{34,139,34}{(+0.19)}}$&$\textbf{70.50}_{\textcolor[RGB]{34,139,34}{(+0.25)}}$&$\textbf{68.71}_{\textcolor[RGB]{34,139,34}{(+0.06)}}$&$\textbf{80.01}_{\textcolor[RGB]{34,139,34}{(+0.19)}}$&$\textbf{28.98}_{\textcolor[RGB]{34,139,34}{(+3.38)}}$&$\textbf{54.68}_{\textcolor[RGB]{34,139,34}{(+0.1)}}$&$\textbf{55.09}_{\textcolor[RGB]{34,139,34}{(+0.17)}}$&$\textbf{49.72}_{\textcolor[RGB]{34,139,34}{(+0.46)}}$\\

  \bottomrule


  \multirow{2}{3.1cm}{\centering Instruct-GPT (175B)\\{\tt{}code-davinci-002}}&SC&78.00&96.78&91.64&100.0&52.01&86.77&81.43&71.58\\
  &SC+\textbf{Self-Verification}&$\textbf{78.32}_{\textcolor[RGB]{34,139,34}{(+0.32)}}$&$\textbf{96.85}_{\textcolor[RGB]{34,139,34}{(+0.07)}}$&$\textbf{92.03}_{\textcolor[RGB]{34,139,34}{(+0.39)}}$&$100.0_{\textcolor[RGB]{34,139,34}{(+0.0)}}$&$\textbf{52.25}_{\textcolor[RGB]{34,139,34}{(+0.24)}}$&$\textbf{86.94}_{\textcolor[RGB]{34,139,34}{(+0.17)}}$&$\textbf{81.53}_{\textcolor[RGB]{34,139,34}{(+0.1)}}$&$\textbf{71.89}_{\textcolor[RGB]{34,139,34}{(+0.31)}}$
   \\

       \bottomrule
     \bottomrule 
\multicolumn{10}{c}{PAL \cite{gao2023pal} For Forward Reasoning}\\
  \bottomrule
       \bottomrule
\multirow{2}{3.1cm}{\centering GPT-3 (175B)\\{\tt{}code-davinci-001}}&PAL&31.82&63.98&63.15&61.52&30.56&42.69&-&-  \\
  &PAL+\textbf{Self-Verification}&$\textbf{32.87}_{\textcolor[RGB]{34,139,34}{(+1.05)}}$&$\textbf{65.45}_{\textcolor[RGB]{34,139,34}{(+1.47)}}$&$\textbf{64.15}_{\textcolor[RGB]{34,139,34}{(+1.0)}}$&$\textbf{61.76}_{\textcolor[RGB]{34,139,34}{(+0.24)}}$&$\textbf{30.90}_{\textcolor[RGB]{34,139,34}{(+0.34)}}$&$\textbf{42.78}_{\textcolor[RGB]{34,139,34}{(+0.09)}}$&-&-\\

  \bottomrule


  \multirow{2}{3.1cm}{\centering Instruct-GPT (175B)\\{\tt{}code-davinci-002}}&PAL&72.02&96.08&92.64&99.15&59.75&79.45&-&-\\
  &PAL+\textbf{Self-Verification}&$\textbf{72.89}_{\textcolor[RGB]{34,139,34}{(+0.87)}}$&$\textbf{96.52}_{\textcolor[RGB]{34,139,34}{(+0.44)}}$&$\textbf{93.78}_{\textcolor[RGB]{34,139,34}{(+1.14)}}$&$\textbf{99.87}_{\textcolor[RGB]{34,139,34}{(+0.72)}}$&$\textbf{60.21}_{\textcolor[RGB]{34,139,34}{(+0.46)}}$&$\textbf{80.24}_{\textcolor[RGB]{34,139,34}{(+0.79)}}$&-&-
   \\
     \bottomrule
       \bottomrule
\end{tabular}}
\caption{Problem solve rate (\%) on reasoning datasets. The previous SoTA results (baselines) are respectively obtained from: (a) GPT-3 175B finetuned \citep{cobbe2021training}; (b) GPT-3 175B finetuned plus an additional 175B verifier \citep{cobbe2021training}; (c) \citet{MinghaoHu2019AMM}; (d) \citet{SubhroRoy2016SolvingGA}; (e) \citet{SubhroRoy2016SolvingGA}; (f) \citet{AidaAmini2019MathQATI}; (g) \citet{XinyuPi2022ReasoningLP}; (h) \citet{YichongXu2022HumanPO}; (i) \citep{cobbe2021training}. In addition, we also attempted to use self-consistency (SC) \citep{XuezhiWangSelfConsistencyIC} or PAL \cite{gao2023pal} (Since this method uses extra programs to replace the results of numerical calculations, we mainly compare it in Arithmetic Tasks.) Decoding to generate candidate answers during the Forward Reasoning stage and combine it with Self-Verification.} 
\label{table2}
\end{table*}

\begin{figure*}[t]
  \centering
  \subfigure[Problem solve rate (\%) in difference size models. The {\tt{}text-ada-001} (0.4B), {\tt{}text-babbage-001} (1.3B), {\tt{}text-curie-001} (7B) and {\tt{}text-davinci-002} (175B) models are used respectively.]{
    \includegraphics[width=0.46\textwidth]{size_zx.pdf}
  }\hspace{7mm}
  \subfigure[Subtract the problem solve rate (\%) of CoT from the problem solve rate (\%) of self-verification in different size models. The pink area means that the use of self-verification has a negative impact.]{
    \includegraphics[width=0.46\textwidth]{size_zx2.pdf}
  }
  \caption{The self-verification ability of models with different sizes.}
    \label{figure4}
\end{figure*}


\section{Experiment Setting}

\subsection{Task and Dataset}
 We evaluated eight datasets on three reasoning tasks: arithmetic reasoning, commonsense reasoning, and logical reasoning. These datasets are highly heterogeneous in terms of their input formats (see Appendix \ref{DD} for the detailed description of each dataset. Examples of different datasets are given in Table \ref{tab:example_table_dataset} of Appendix \ref{AER}).

 \begin{itemize}
    \item \textbf{Arithmetic Reasoning.} We performed experiments on the following 6 arithmetic datasets: SingleEq \citep{RikKoncelKedziorski2015ParsingAW}, AddSub \citep{MohammadJavadHosseini2014LearningTS}, MultiArith \citep{SubhroRoy2016SolvingGA}, AQUA-RAT \citep{WangLing2017ProgramIB}, GSM8K \citep{cobbe2021training}, and SVAMP \cite{PatelArkil2021AreNM}.
    \item \textbf{Commonsense Reasoning.} CommonsenseQA (CSQA) \citep{AlonTalmor2018CommonsenseQAAQ} is the most typical dataset of the task, which requires commonsense knowledge about the world to accurately answer questions with complex meanings.
    \item \textbf{Logical Reasoning.} Date Understanding (DU) \cite{AarohiSrivastava2022BeyondTI} involves inferring a date from a given context.
\end{itemize}

\subsection{Model}

We conducted experiments to evaluate the original GPT-3 \citep{DBLP:journals/corr/abs-2107-03374} ({\tt{}code-davinci-001}) model and the Instruct-GPT model~\citep{LongOuyang2022TrainingLM} ({\tt{}code-davinci-002}). Additionally, we conducted analysis experiments with public GPT-3 \citep{brown2020language}. All prediction results of different reasoning tasks and datasets are obtained by OpenAI's API~\footnote{OpenAI's API: \url{https://openai.com/api/}}. Appendix \ref{RS} shows the reproducibility statement.

\subsection{Prompts}

We conducted all experiments in the few-shot setting without any fine-tuning of the original LLM To ensure a fair comparison, we used the same prompts as in \citet{wei2022chain} for forward reasoning. We made several changes of the prompts for backward verification (the details are shown in Appendix \ref{Prompt}).




\subsection{Implementation}

In each experiment, we perform CoT prompting on the LLMs, then LLMs generate conclusions (answers) by sampling decoding without top-k truncation. When forward reasoning, we generated $K=5$ candidate answers (conclusions). In backward verification, each candidate conclusion generated $P=10$ times, and the maximum token length of each decoding was 168. After LLM generates the output, we only select the part of the text that conforms to the conclusion format. Appendix \ref{answercleansing} shows the specific strategy for different tasks. In addition, to ensure a fair comparison, we ran each experiment three times and calculated the average result.


\section{Result and Analysis}

The main experimental results are shown in Table \ref{table2}. The table shows that the proposed self-verification method (SV) can improve previous methods in all datasets. Our method achieved a new state-of-the-art (SOTA) performance in six of these eight datasets. Appendix \ref{AER} shows specific examples of language model self-verification for each dataset. Additionally, we observed that self-verification led to an average increase of 2.33\% in the high-performing Instruct-GPT model, which indicates that the model with strong forward reasoning capabilities also benefits from the self-verification mechanism. The detailed experimental conclusions and analysis are described as follows: 


\begin{figure}[t]
  \begin{center}
    \includegraphics[width=0.5\textwidth]{few_zx.pdf}
  \end{center}
  \caption{Problem solve rate (\%)  comparison of 2-shot to 8-shot prompts.}
  \label{6}
\end{figure}
\begin{figure*}[t]
\begin{center}
	\includegraphics[scale=0.238]{multi_single_zx.pdf}
\end{center}
\caption{Comparison of problem solve rate (\%) between single-condition verification and multiple-condition verification.}
\label{123}
\end{figure*}

\begin{figure*}[t]
\begin{center}
	\includegraphics[scale=0.183]{judge.pdf}
\end{center}
\caption{Comparison of problem solve rate (\%) for the ``CMV'' and the ``TFV'' in arithmetic tasks.}
\label{16}
\end{figure*}

\textbf{The current self-verification method is more suitable for arithmetic reasoning tasks than other reasoning tasks.} We find that the average performance improvement of arithmetic reasoning tasks ($1.67\%/2.84\% \uparrow$) is higher than that of other reasoning tasks ($0.62\%/0.78\% \uparrow$) in Table \ref{table2}. We believe the reason is that it is easier to find the required mask conditions for arithmetic reasoning tasks, but other reasoning tasks used TFV that cannot determine the exact conditions. In the future, we will consider the targeted condition selection and masking for other reasoning tasks. 

\textbf{The self-validation method can be combined with improved methods for forward reasoning.} We report the results of combining self-consistency or PoL at the bottom of Table \ref{table2} separately. Specifically, for combining self-consistency, we use the Top-2 candidate results obtained from self-consistency in the Forward Reasoning stage and then use self-validation to re-rank the candidate results; for combining PAL, we require the generation of runnable programs in Forward Reasoning to obtain candidate answers. We find that this approach still can achieve better performance than self-consistency, demonstrating that self-verification can be combined with a series of existing methods for improving forward calculation to achieve further gains. We believe that the self-verification can re-rank candidate answers from the perspective of backward validation, providing more robust results.


\textbf{Larger language models are better reasoners with self-verification.}  Figure \ref{figure4} shows the capability of GPT-3 models with parameters ranging from 0.4B to 175B. The experimental results suggest that the self-verification ability tends to be more robust as the number of parameters increases, aligning with the limited experimental results in \citet{wei2022chain}. This indicates that self-verification is an emergent property that arises in larger models, where stronger reasoning capacities allow them to derive reliable and accurate conclusions during the context learning process \citep{ho2022large, wang2023large}. Consequently, their performance in the backward verification phase is also enhanced. However, smaller models are prone to generate erroneous text during the generation process, and augment them with self-verification abilities presents a challenge.




\textbf{With the different number of few-shots, the reasoning ability of models using self-verification has significantly improved.} Figure \ref{6} demonstrate the impact of different sample sizes on three arithmetic reasoning datasets. We observe that the self-verification method exhibits greater robustness with smaller samples, even with only 2-shots (At this time, it has 99.6\% performance of 8-shot, while CoT has only 98.7\%). In addition, we find that even with only four samples (2 CoT samples + 2 self-verification samples), self-verification outperforms the CoT with eight samples, which highlights the importance of answer verification in scenarios of limited data.







\textbf{The more verification conditions are used, the better self-verification reasoning ability.} We observed the effect of using the single conditional mask on six different arithmetic datasets for Condition Mask Verificat in Figure \ref{123}. As each number in these datasets' input can be thought of as a condition, we can study the impact of increasing the number of validation conditions. In most experiments, we found that the multi-condition mask performed better than the single-condition mask, and both performed better than the original CoT. These results suggest that the accuracy of verification scores improves as the number of available conditions increases.
\begin{figure}[h]
  \centering


    \includegraphics[width=0.46\textwidth]{jszy.pdf}
  
  \caption{The computational resource of the proposed method on GSM8K.}
    \label{figure12}
\end{figure}


\textbf{Masked conditions can guide the LLMs to reason more effectively.} As shown in Figure \ref{16}, we compared the results of using CMV (Conditional Masked Verification) and TFV (Token Form Verification) for self-verification. We found that the performance of CMV is generally better than TFV. We believe this is because the lack of explicit goals can lead to a lack of use of existing conclusions, so CMV is more helpful in stimulating the self-verification ability of the model. However, due to its simplicity, TFV can be applied to a variety of tasks (including common sense reasoning and logical reasoning, both with improvements compared to the CoT baseline) for self-verification, making it highly adaptable to different scenarios.

\textbf{Fewer computational resources can also improve performance through self-verification.} In Figure \ref{figure12}, we show the results of changing the number of $P$ generated in Backward Verification. We find that even when $P=2$, only a small increase in computational overhead is needed, and there is still an improvement in CoT baseline. Considering that performance starts to slowly increase when P is increased to 10, we recommend choosing an appropriate value for P (e.g. P=10) to achieve a balance between performance and resource consumption.


\begin{table}[h]
\centering
\resizebox{0.48\textwidth}{!}{
\begin{tabular}{lccrrr}
\hline
\textbf{Dataset} & \textbf{Num.} & \textbf{Avg. Ans.} & \textbf{Ran. Acc.} & \textbf{CoT Acc.} & \textbf{Veri. Acc.} \\
\hline
GSM8K & 601/1306 & 2.80 & 35.7\% & 53.4\% & \textbf{58.9\%} \\
Addsub & 77/377 & 2.13 & 46.9\% & 56.7\% & \textbf{74.0\%} \\
AQUA & 123/219 & 2.71 & 36.9\% & 50.3\% & \textbf{51.2\%} \\
Multiarith & 73/593 & 2.19 & 45.6\% & 71.2\% & \textbf{89.0\%} \\
SingleEq & 51/501 & 2.31 & 43.3\% & 51.0\% & \textbf{74.5\%} \\
SVAMP & 274/954 & 2.54 & 39.4\% & 51.5\% & \textbf{58.4\%} \\
\hline
\end{tabular}}

\caption{Further analysis of the experimental results in Table \ref{table2}. Where ``Num.'' represents the total number of samples in this setting and the total number of original dataset samples, ``Avg. Ans.'' means the average number of candidate answers, ``Ran. Acc.'' = 1 / ``Avg. Ans.''. ``CoT Acc.'' refers to the accuracy of the CoT method when temperature = 0. And ``Veri. Acc.'' refers to the probability of picking out the correct answer from the candidate answers in backward verification stage.}
\label{table:analy}
\end{table}

\textbf{Analysis of the accuracy of the verification results.} We conducted additional analysis of the InstructGPT results in Table \ref{table2} under a new setting where we only considered cases in the forward inference phase that contained one correct answer and $N$ incorrect answers (where $N$>=1 and $N$<=4). The results in Table \ref{table:analy} provide further evidence that the proposed self-verification technique can effectively improve the accuracy of commonsense reasoning models. Across all 6 datasets, the verification accuracy is consistently and considerably higher than both the random guessing baseline and the standalone CoT model accuracy. For example, on the challenging GSM8K dataset, the verification stage obtains 58.9\% accuracy, substantially outperforming the 53.4\% CoT accuracy and 35.7\% random guess accuracy. The largest accuracy gains are witnessed on the MultiArith and SingleEq datasets, where the verification stage lifts the accuracy by 17.8\% and 23.5\% respectively over the CoT model. This indicates that the self-verification technique is particularly adept at rectifying errors made by the CoT model on arithmetic and symbolic equation problems. The consistent accuracy improvements demonstrate that allowing the model to verify its own predictions provides a simple yet effective way to enhance commonsense reasoning. These comprehensive results validate self-verification as a promising approach to refine LLMs and reduce reasoning errors.


\section{Conclusion}
In this study, we show that large language models have a strong ability to self-verification, allowing them to assess the conclusions they generate accurately. We propose a novel method that uses self-verification to generate interpretable scores for ranking results in few-shot tasks. Our approach demonstrates the potential of using self-verification to improve the accuracy and reliability of large language models in reasoning tasks. By relying on the self-verification ability of large language models, we significantly improved the accuracy of three types of reasoning tasks. All in all, we posit that the self-verification capability of large language models will have an extensive positive impact, as it enables the enhancement of their reasoning abilities through a simple process of self-verification.


\section*{Limitations}
Our self-verification method relies on large language models (LLMs). It provides few-shot prompts to guide the model in verifying its own results, but it is worth noting that these prompts are artificially constructed and may introduce bias. The effectiveness of our method is limited by the presence of accurate answers within the candidate conclusions generated by the LLM, our experiments demonstrate that the capability of self-verification depends on the model's reasoning abilities, which means it is challenging to augment the reasoning performance of smaller language models, while the benefits are greater for high-performing models (in Figure \ref{figure4}). For the sake of usability, our method focuses on the conclusions derived from the reasoning rather than the reasoning process itself, and is thus not suitable for evaluating the LLM's inference procedure. Additionally, the method necessitates generating multiple candidate inference chains and conclusions, leading to increased computational costs; however, we demonstrate that only a minimal increase (merely 1x) in expenditure is required to substantially enhance the model's inference capabilities (in Figure \ref{figure12}).

\section*{Acknowledgements}
This work was supported by the National Key R\&D Program of China (No.2022ZD0118501) and the National Natural Science Foundation of China
(No.62376270, No.U1936207, No.61976211, No.62171183). This work was supported by the Strategic Priority Research Program of Chinese Academy of Sciences (No.XDA27020100),  Youth Innovation Promotion Association CAS, and OPPO Research Fund.
