Online SFT for LLM Reasoning

Mengqi Li, Lei Zhao, Anthony Man-Cho So, Ruoyu Sun, Xiao Li

Keywords: LLM reasoning, reasoning model

Abstract:

We present a simple, self-help online supervised finetuning (**OSFT**) method for LLM reasoning. In this paradigm, the model generates its own responses and is immediately finetuned on this self-generated data. OSFT is a highly efficient training strategy for LLM reasoning, as it is *reward-free* and uses just one rollout by default.
Experiment results show that OSFT achieves downstream performance on challenging mathematical reasoning tasks comparable to strong reinforcement learning (RL) methods such as GRPO. Our ablation study further demonstrates the efficiency and robustness of OSFT. The major mechanism of OSFT lies in facilitating the model's own existing preference (latent knowledge) learned from pretraining, which leads to reasoning ability improvement.
We believe that OSFT offers an efficient and promising alternative to more complex, reward-based training schemes.

Primary Area: foundation or frontier models, including LLMs

Submission Number: 2724