Why Distillation can Outperform Zero-RL: The Role of Flexible Reasoning

ICLR 2026 Conference Submission12645 Authors

Keywords: LLM, reasoning, distillation, zero-RL

TL;DR: We find that distillation with a small number of samples can endow the base model with flexible reasoning patterns, which are crucial for tackling complex reasoning tasks.

Abstract: Reinforcement learning (RL) has played an important role in improving the reasoning ability of large language models (LLMs). Some studies apply RL directly to \textit{smaller} base models (known as zero-RL) and also achieve notable progress. However, in this paper, we show that using only 920 examples, a simple distillation based on the base model can clearly outperform zero-RL, which typically requires much more data and computational cost. By analyzing the token frequency in model outputs, we find that the distilled model shows more flexible reasoning. It uses anthropomorphic tokens and logical connectors much more often than the zero-RL model. Further analysis reveals that distillation enhances the presence of two advanced cognitive behaviors: Multi-Perspective Thinking or Attempting and Metacognitive Awareness. Frequent occurrences of these two advanced cognitive behaviors give rise to flexible reasoning, which is essential for solving complex reasoning problems.

Primary Area: foundation or frontier models, including LLMs

Submission Number: 12645