Language Models That Think, Chat Better

ICLR 2026 Conference Submission20534 Authors

Keywords: language model, GRPO, DPO, PPO, thinking, chat, test time scaling, RLHF, RLVR

TL;DR: We teach language models to think before answering to improve them on general tasks.

Abstract:

Reinforcement learning with verifiable rewards (RLVR) improves language model
reasoning by using rule-based rewards in verifiable domains such as mathematics and code. However, RLVR leads to limited generalization for open-ended
tasks—such as writing outline essays or making meal plans—where humans reason routinely. This paper shows that the RLVR paradigm is effective beyond verifiable domains, and introduces **RL** with **M**odel-rewarded **T**hinking (RLMT) for
general-purpose chat capabilities. Using diverse real-world prompts, RLMT requires LMs to generate long CoT reasoning before response, and optimizes them
with online RL against a preference-based reward model used in RLHF. Across
40 training runs on Llama-3.1-8B and Qwen-2.5-7B (both base and instruct) and
multiple optimization algorithms (DPO, PPO, and GRPO), RLMT consistently
outperforms standard RLHF pipelines. This includes substantial gains of 3–7
points on three chat benchmarks (AlpacaEval2, WildBench, and ArenaHardV2),
along with 1–3 point improvements on other tasks like creative writing and general knowledge. Our best 8B model surpasses GPT-4o in chat and creative writing and rivals Claude-3.7-Sonnet (Thinking). RLMT can also be applied directly
to base models without an SFT stage, akin to R1-Zero training (DeepSeek-AI,
2025). Remarkably, with only 7K prompts, Llama-3.1-8B base trained with our
RLMT recipe outperforms Llama-3.1-8B-Instruct post-trained with a complex
multi-staged pipeline with 25M+ examples. We close with qualitative and quantitative analyses of how trained models plan their responses. Our results rethink
the post-training pipeline and call upon future work to understand and employ
thinking more broadly

Primary Area: foundation or frontier models, including LLMs

Submission Number: 20534