RL of Thoughts: Navigating LLM Reasoning with Inference-time Reinforcement Learning

ICLR 2026 Conference Submission15018 Authors

Keywords: Large language models, reasoning, reinforcement learning

TL;DR: This paper proposed to utilize RL at inference-time to navigate the reasoning logics of LLMs, which significantly enhances the reasoning capabilities of multiple LLMs across various tasks with less than 3K parameters in the RL navigator model.

Abstract:

Despite rapid advancements in large language models (LLMs), the token-level autoregressive nature constrains their complex reasoning capabilities.
To enhance LLM reasoning, inference-time techniques, including Chain/Tree/Graph-of-Thought(s), successfully improve the performance, as they are fairly cost-effective by guiding reasoning through external logical structures without modifying LLMs' parameters.
However, these manually predefined, task-agnostic frameworks are applied uniformly across diverse tasks, lacking adaptability.
To improve this, we propose **RL-of-Thoughts (RLoT)**, where we train a lightweight navigator model with reinforcement learning (RL) to generate task-adaptive logical structures at inference time, enhancing LLM reasoning.
Specifically, we design five basic logic blocks from the perspective of human cognition.
During the reasoning process, the trained RL navigator dynamically selects the suitable logic blocks and combines them into task-specific logical structures according to problem characteristics.
Experiments across multiple reasoning benchmarks (AIME, MATH, GPQA, etc.) with multiple LLMs (GPT, Llama, Qwen, and DeepSeek) illustrate that RLoT outperforms established inference-time techniques by up to 13.4\%.
Remarkably, with less than 3K parameters, our RL navigator is able to make sub-10B LLMs comparable to 100B-scale counterparts.
Moreover, the RL navigator demonstrates strong transferability: a model trained on one specific LLM-task pair can effectively generalize to unseen LLMs and tasks.
Our code is open-source at https://anonymous.4open.science/r/RL-LLM-Reasoning-1A30.

Primary Area: reinforcement learning

Submission Number: 15018