GuardAgent: Safeguard LLM Agent by a Guard Agent via Knowledge-Enabled Reasoning

Zhen Xiang, Linzhi Zheng, Yanjie Li, Junyuan Hong, Qinbin Li, Han Xie, Jiawei Zhang, Zidi Xiong, Chulin Xie, Carl Yang, Dawn Song, Bo Li

Keywords: LLM agent, guardrail, safety, LLM

TL;DR: We propose GuardAgent, the first LLM agent as a guardrail to protect other LLM agents via knowledge-enabled reasoning.

Abstract:

The rapid advancement of large language models (LLMs) has catalyzed the deployment of LLM-powered agents across numerous applications, raising new concerns regarding their safety and trustworthiness. In addition, existing methods for enhancing the safety of LLMs are not directly transferable to LLM-powered agents due to their diverse objectives and output modalities. In this paper, we propose
GuardAgent, the first LLM agent as a guardrail to protect other LLM agents.
Specifically, GuardAgent oversees a target LLM agent by checking whether its
inputs/outputs satisfy a set of given guard requests, e.g., safety rules or privacy
policies defined by the users. The pipeline of GuardAgent consists of two steps: 1) create a task plan by analyzing the provided guard requests, and 2) generate
guardrail code based on the task plan and execute the code by calling APIs or
using external engines. In both steps, an LLM is utilized as the core reasoning
component, supplemented by in-context demonstrations retrieved from a memory
module storing information from previous sessions. Such knowledge-enabled reasoning of GuardAgent allows it to understand various textual guard requests and
accurately “translate” them into executable code that provides reliable guardrails.
Furthermore, GuardAgent is equipped with an extendable toolbox containing
relevant APIs and functions, and requires no additional LLM training, underscoring
its flexibility and low operational overhead. In addition to GuardAgent, we
propose two novel benchmarks: an EICU-AC benchmark for assessing privacy-
related access control for healthcare agents and a Mind2Web-SC benchmark for
assessing safety regulations for web agents. When using Llama3-70B/Llama3.1-
70B/GPT-4 as the core LLM, GuardAgent achieves 98.4%/98.4%/98.7% and
83.5%/84.5%/90.0% guarding accuracy on these two benchmarks in moderating
invalid inputs and outputs of two types of agents, respectively. We also show the
ability of GuardAgent to define necessary functions that are absent from the
toolbox, which further highlights the flexibility of GuardAgent in adaption to
new LLM agents and guard requirements.

Primary Area: foundation or frontier models, including LLMs

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.

Submission Guidelines: I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.

Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.

No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.

Submission Number: 9833