SIRI: Scaling Iterative Reinforcement Learning with Interleaved Compression

ICLR 2026 Conference Submission14725 Authors

Keywords: LLM, RL, efficiency

Abstract:

We introduce SIRI, **S**caling **I**terative **R**einforcement Learning with **I**nterleaved Compression, a simple yet effective RL approach for Large Reasoning Models (LRMs) that enables more efficient and accurate reasoning. Existing studies have observed repetitive thinking patterns in LRMs, and attempt to reduce them at the cost of performance. In this paper, we show that this trade-off can be overcome through a training regime that iteratively alternates between compressing and expanding the reasoning budget, by dynamically adjusting the maximum rollout length during training. The *compression phase* cuts the rollout length, forcing the model to make precise and valuable decisions in limited context, which effectively reduces redundant tokens and increases reasoning density. The *expansion phase* then relaxes the length limit, providing space for the model to explore and plan in long-horizon settings.
Remarkably, we find that after each compression–expansion cycle, the model’s performance improves even as its output length decreases, steadily pushing it closer to the Pareto frontier in the performance–efficiency trade-off.
Training on DeepSeek-R1-Distill-Qwen-1.5B, SIRI-low improves performance on AIME24 by 43.2\% while reducing token usage by 46.9\% after three iterations, and SIRI-high achieves the highest accuracy
compared to all other methods (Figure 1).
Our findings shed light on the potential of periodically oscillating the LRM's output truncation length during training to dynamically balance exploration and efficiency in reasoning, converging towards an optimal "sweet spot" between the two.

Primary Area: foundation or frontier models, including LLMs

Submission Number: 14725