VeriGuard: Enhancing LLM Agent Safety via Verified Code Generation

ICLR 2026 Conference Submission15990 Authors

Keywords: agent security, llm security, code verification, security policies generation, safety

Abstract:

The deployment of autonomous AI agents in sensitive domains, such as healthcare, introduces critical risks to safety, security, and privacy. These agents may deviate from user objectives, violate data handling policies, or be compromised by adversarial attacks. Mitigating these dangers necessitates a mechanism to formally guarantee that an agent's actions adhere to predefined safety constraints, a challenge that existing systems do not fully address.
We introduce \ours{}, a novel framework that provides formal safety guarantees for LLM-based agents through a dual-stage architecture designed for robust and verifiable correctness. The initial offline stage involves a comprehensive validation process.
It begins by clarifying user intent to establish precise safety specifications. \ours{} then synthesizes a behavioral policy and subjects it to both extensive testing in simulated environments and rigorous formal verification to mathematically prove its compliance with these specifications.
This iterative process refines the policy until it is deemed correct. Subsequently, the second stage provides online action monitoring, where \ours{} operates as a runtime monitor to validate each proposed agent action against the pre-verified policy before execution. This separation of the exhaustive offline validation from the lightweight online monitoring allows formal guarantees to be practically applied, providing a robust safeguard that substantially improves the trustworthiness of LLM agents in complex, real-world environments.

Primary Area: alignment, fairness, safety, privacy, and societal considerations

Submission Number: 15990