Reinforced Preference Optimization for Recommendation

ICLR 2026 Conference Submission17548 Authors

Keywords: Recommender System, Large Language Model, Reinfocement Learning with Verifiable Reward

TL;DR: Our paper proposes ReRe, a novel reinforcement fine-tuning paradigm tailored for LLM-based recommenders by adapting the sampling strategy and the reward.

Abstract:

Recent breakthroughs in large language models (LLMs) have fundamentally shifted recommender systems from discriminative to generative paradigms, where user behavior modeling is achieved by generating target items conditioned on historical interactions.
Yet current generative recommenders still suffer from two core limitations: the lack of high-quality negative modeling and the reliance on implicit rewards.
Reinforcement learning with verifiable rewards (RLVR) offers a natural solution by enabling on-policy sampling of harder negatives and grounding optimization in explicit reward signals.
However, applying RLVR to generative recommenders remains non-trivial.
Its unique generation space often leads to invalid or repetitive items that undermine sampling efficiency, and ranking supervision is sparse since most items receive identical zero rewards.
To address these challenges, we propose \textbf{Reinforced Preference Optimization for Recommendation} (\textbf{ReRe}), a reinforcement-based paradigm tailored to LLM-based recommenders, an important direction in generative recommendation.
ReRe incorporates constrained beam search to improve sampling efficiency and diversify hard negatives, while augmenting rule-based accuracy rewards with auxiliary ranking rewards for finer-grained supervision.
Extensive experiments on three real-world datasets demonstrate that ReRe consistently outperforms both traditional and LLM-based recommenders in ranking performance.
Further analysis shows that ReRe not only enhances performance across both base and SFT-initialized models but also generalizes robustly across different backbone families and scales.
Beyond empirical gains, we systematically investigate the design space of RLVR in recommendation across generation, sampling strategy, reward modeling, and optimization algorithm, offering insights for future research.
Our codes are available at \url{https://anonymous.4open.science/r/ReRe-E1B0}.

Primary Area: other topics in machine learning (i.e., none of the above)

Submission Number: 17548