Let’s Try Again: Eliciting Multi-Turn Reasoning in Language Models via Simplistic Feedback

Licheng Liu, Zihan Wang, Linjie Li, Chenwei Xu, Yiping Lu, Han Liu, Avirup Sil, Manling Li

Keywords: Reinforcement Learning, Large Reasoning Model

Abstract:

Multi-turn problem solving is a critical yet challenging scenario in the practical application of Large Reasoning Models (LRMs), commonly encountered in domains such as chatbots, programming assistants, and education. Recently, reasoning models like DeepSeek-R1 has shown the promise of reinforcement learning (RL) methods in enhancing model reasoning capabilities. However, we observe that models trained with existing single-turn RL paradigms often lose their ability to solve problems across multiple turns, struggling to revise answers based on context and exhibiting repetitive responses. This raises new challenges in preserving reasoning abilities while enabling multi-turn contextual adaptation.
In this work, we find that simply allowing models to engage in multi-turn problem solving where they receive only unary feedback (e.g., “Let’s try again”) after incorrect answers can help recover both single-turn and interactive multi-turn reasoning skills. We introduce Unary Feedback as Observation (UFO) for reinforcement learning, a method that explicitly leverages minimal yet natural user feedback during iterative problem-solving which can be easily applied to any existing single-turn RL training paradigm. Experimental results show that RL training with UFO preserves single-turn performance while improving multi-turn reasoning accuracy by 14%, effectively utilizing sparse feedback signals when available. To further reduce superficial guessing and encourage comprehensive reasoning, we explore reward structures that incentivize thoughtful, deliberate answers across interaction turns. Code and models will be publicly released.

Submission Number: 36