Beyond Token-Level Policy Gradients for Complex Reasoning with Large Language Models

ACL ARR 2026 January Submission2983 Authors

Keywords: Reinforcement Learning, Reasoning

Abstract:

Existing policy-gradient methods for auto-regressive language models typically selects subsequent tokens one at a time as actions in the policy.
While effective for many generation tasks, such an approach may not fully capture the structure of complex reasoning tasks, where a single semantic decision is often realized across multiple tokens—for example, when defining variables or composing equations. This introduces a potential mismatch between token-level optimization and the inherently block-level nature of reasoning in these settings.
To bridge this gap, we propose Multi-token Policy Gradient Optimization (MPO), a framework that treats sequences of K consecutive tokens as unified semantic actions.
This block-level perspective enables our method to capture the compositional structure of reasoning trajectories and supports optimization over coherent, higher-level objectives.
Experiments on mathematical reasoning and coding benchmarks show that MPO outperforms standard token-level policy gradient baselines, highlight the limitations of token-level policy gradients for complex reasoning, motivating future research to look beyond token-level granularity for reasoning-intensive language tasks.

Paper Type: Long

Research Area: Machine Learning for NLP

Research Area Keywords: reinforcement learning, reasoning

Contribution Types: NLP engineering experiment

Languages Studied: English

Submission Number: 2983