Synthetic Error Injection Fails to Elicit Self-Correction In Language Models

David Xing Wu, Shreyas Kapur, Anant Sahai, Stuart Russell

Keywords: Supervised Fine-Tuning, Reinforcement Learning, Self-Correction, LLMs, Reasoning

TL;DR: Directly finetuning models to self-correct with synthetic errors is significantly hampered by distribution shift in the errors.

Abstract: Reasoning language models trained with reinforcement learning have been observed to exhibit emergent self-correction behavior. Inspired by the success of synthetic error injection in autonomous driving and robotics, we attempt to directly teach models to self-correct without reinforcement learning. Our approach inserts artificial errors into reasoning chains, masks them, and supervises the model to recognize and correct these mistakes. Despite the intuitive appeal of this method, we find that it fails to significantly improve performance even on simple synthetic tasks across multiple models. Moreover, even when the model catches its own error, it often parrots the original mistake. We find that the distribution shift of synthetic errors to on-policy errors significantly degrades the error-correction capabilities of the fine-tuned model, even with good synthetic coverage of on-policy errors. Our results help explain why on-policy reinforcement learning methods have proven uniquely effective for eliciting self-correction.

Email Sharing: We authorize the sharing of all author emails with Program Chairs.

Data Release: We authorize the release of our submission and author names to the public in the event of acceptance.

Submission Number: 85