Agent Safety Alignment via Reinforcement Learning

ACL ARR 2026 January Submission3095 Authors

Keywords: Agent; Large Lanuage Models; AI Security

Abstract:

The emergence of autonomous Large Language Model (LLM) agents capable of tool usage has introduced new safety risks that go beyond traditional conversational misuse.
These agents, empowered to execute external functions, are vulnerable to both user-initiated threats (e.g., adversarial prompts) and tool-initiated threats (e.g., malicious outputs from compromised tools).
In this paper, we propose the first unified safety-alignment framework for tool-using agents, enabling models to handle both channels of threat via structured reasoning and sandboxed reinforcement learning.
We introduce a three-way taxonomy that classifies both user prompts and tool responses as benign, malicious, or sensitive, and we define a policy-driven decision model to train the agents.
Our framework employs a custom-designed sandbox environment that simulates real-world tool execution and allows fine-grained reward shaping.
Through extensive evaluations on public and self-built benchmarks, including Agent SafetyBench, InjecAgent, and BFCL, we demonstrate that our safety-aligned agents significantly improve resistance to security threats while preserving strong utility on benign tasks.
Our results show that safety and effectiveness can be jointly optimized, laying the groundwork for trustworthy deployment of autonomous LLM agents.

Paper Type: Long

Research Area: Safety and Alignment in LLMs

Research Area Keywords: Language Modeling

Contribution Types: Model analysis & interpretability, NLP engineering experiment

Languages Studied: English

Submission Number: 3095