Improving the Language Understanding Capabilities of Large Language Models Using Reinforcement Learning

ACL ARR 2025 May Submission3950 Authors

Abstract: Instruction-fine-tuned large language models (LLMs) under 14B parameters continue to underperform on natural language understanding (NLU) tasks, often trailing smaller models like BERT-base on benchmarks such as GLUE and SuperGLUE. Motivated by the success of reinforcement learning in reasoning tasks (e.g., DeepSeek), we explore Proximal Policy Optimization (PPO) as a framework to improve the NLU capabilities of LLMs. We frame NLU as a reinforcement learning environment, treating token generation as a sequence of actions and optimizing for reward signals based on alignment with ground-truth labels. PPO consistently outperforms supervised fine-tuning, yielding an average improvement of 6.3 points on GLUE, and surpasses zero-shot and few-shot prompting by 38.7 and 26.1 points, respectively. Notably, PPO-tuned models outperform GPT-4o by over 4\% on average across sentiment and natural language inference tasks, including gains of 7.3\% on the Mental Health dataset and 10.9\% on SIGA-nli. This work highlights a promising direction for adapting LLMs to new tasks by reframing them as reinforcement learning problems, enabling learning through simple end-task rewards rather than extensive data curation.

Paper Type: Long

Research Area: Machine Learning for NLP

Research Area Keywords: Natural Language Understanding (NLU), Large Language Models (LLMs), Reinforcement Learning (RL), Proximal Policy Optimization (PPO)

Contribution Types: NLP engineering experiment, Position papers

Languages Studied: English

Submission Number: 3950