PoLi-RL: A Point-to-List Reinforcement Learning Framework for Conditional Semantic Textual Similarity

ICLR 2026 Conference Submission18363 Authors

Keywords: Conditional Semantic Textual Similarity, Reinforcement Learning, Large Language Models, Natural Language Processing, Curriculum Learning

TL;DR: We propose PoLi-RL, a two-stage RL framework that stabilizes listwise optimization for the C-STS task via a progressive curriculum and a Parallel Slice Ranking Reward mechanism , achieving SOTA performance for the cross encoder architecture.

Abstract: Conditional Semantic Textual Similarity (C-STS) measures the semantic proximity between text segments under a specific condition, thereby overcoming the ambiguity inherent in traditional STS. However, existing methods are largely confined to discriminative models, failing to fully integrate recent breakthroughs in the NLP community concerning Large Language Models (LLMs) and Reinforcement Learning (RL). RL is a particularly well-suited paradigm for this task, as it can directly optimize the non-differentiable Spearman ranking metric and guide the reasoning process required by C-STS. However, we find that naively applying listwise RL leads to unstable training and convergence failure, as the model is overwhelmed by a complex, sparse reward signal. To address this challenge, we introduce PoLi-RL, a novel Point-to-List Reinforcement Learning framework. PoLi-RL employs a two-stage curriculum: it first trains the model with simple pointwise rewards to establish fundamental scoring capabilities, then transitions to a hybrid reward that combines pointwise, pairwise, and listwise objectives to refine the model's ability to discern subtle semantic distinctions. Crucially, we propose an innovative Parallel Slice Ranking Reward (PSRR) mechanism that computes ranking rewards in parallel slices, where each slice comprises same-indexed completions from different samples. This provides precise, differentiated learning signals that ensure training stability. On the official C-STS benchmark, PoLi-RL achieves a Spearman correlation coefficient of 48.18, establishing a new SOTA for the cross-encoder architecture. As the first work to successfully apply RL to C-STS, our study introduces a powerful and stable paradigm for training LLMs on complex, ranking-based conditional judgment tasks. Our code and checkpoints are available at https://anonymous.4open.science/r/PoLi-RL.

Primary Area: applications to computer vision, audio, language, and other modalities

Submission Number: 18363