Benchmarking the Robustness of Agentic Systems to Adversarially-Induced Harms

Jonathan Nöther, Adish Singla, Goran Radanovic

Keywords: Agentic Systems, Multi-Agent Systems, Adversarial Machine Learning

TL;DR: We propose a benchmark for agentic systems with regards to harmful actions and analyze the robustness of agentic systems against attackers.

Abstract:

Ensuring the safe use of agentic systems requires a thorough understanding of the range of malicious behaviors these systems may exhibit when under attack. In this paper, we evaluate the robustness of LLM-based agentic systems against attacks that aim to elicit harmful actions from agents. To this end, we propose a novel taxonomy of harms for agentic systems and a novel benchmark, BAD-ACTS, for studying the security of agentic systems with respect to a wide range of harmful actions. BAD-ACTS consists of five implementations of agentic systems in distinct application environments, as well as a dataset of 238 high-quality examples of
harmful actions and an extended dataset containing 699 additional adversarial actions. This enables a comprehensive study of the robustness of agentic systems across a wide range of categories of harmful behaviors, available tools, and inter-agent communication structures. Using this benchmark, we analyze the robustness of agentic systems against an attacker that controls one of the agents in the system and aims to manipulate other agents to execute a harmful target action. Our results show that the attack has a high success rate, demonstrating that even a single adversarial agent within the system can have a significant impact on the security. This attack remains effective even when agents use a simple prompting-based defense strategy. However, we additionally propose a more effective defense based
on zero-shot message monitoring. We believe that this benchmark provides a
diverse testbed for the security research of agentic systems.

Primary Area: datasets and benchmarks

Submission Number: 13947