Beyond Introspection: Reinforcing Thinking via Externalist Behavioral Feedback

Diji Yang, Linda Zeng, Kezhen Chen, Yi Zhang

Keywords: Novel paradigms for generalization, Learning or incorporating feedback models, Test-time strategies that best support reasoning for an LM

TL;DR: We propose DRR, a three-step framework that distills an LLM’s own reasoning into synthetic behavioral data and trains a lightweight discriminative model to improve the inference-time thinking without relying on human-labeled reasoning steps.

Abstract: While inference-time thinking allows Large Language Models (LLMs) to address complex problems, the extended thinking process can be unreliable or inconsistent because of the model's probabilistic nature, especially near its knowledge boundaries. Existing approaches attempt to mitigate this by having the model critique its own reasoning to make corrections. However, such self-critique inherits the same biases of the original output, known as the introspection illusion. Moving beyond such introspection and inspired by core methodologies in ethology, we propose an externalist three-step framework \textbf{D}istillation-\textbf{R}einforcement-\textbf{R}easoning (\textbf{DRR}). Rather than relying on a model's introspection, DRR evaluates its observable behaviors to provide corrective feedback. DRR first distills the reasoner's behavioral traces, then trains a lightweight, external Discriminative Model (DM). At inference time, this DM acts as a critic, identifying and rejecting suspicious reasoning steps. This external feedback compels the LLM to discard flawed pathways and explore alternatives, thereby enhancing reasoning quality without altering the base model. Experiments on multiple reasoning benchmarks show that our framework significantly outperforms prominent self-critique methods. Benefiting from a lightweight and annotation-free design, DRR offers a scalable and adaptable solution for improving the reliability of reasoning in a wide range of LLMs.

Submission Number: 87