EnIGMA: Interactive Tools Substantially Assist LM Agents in Finding Security Vulnerabilities

Talor Abramovich, Meet Udeshi, Minghao Shao, Kilian Lieret, Haoran Xi, Kimberly Milner, Sofija Jancheska, John Yang, Carlos E Jimenez, Farshad Khorrami, Prashanth Krishnamurthy, Brendan Dolan-Gavitt, Muhammad Shafique, Karthik R Narasimhan, Ramesh Karri, Ofir Press

TL;DR: We develop an LM agent for solving Capture The Flag security challenges, introducing Interactive Agent Tools such as a debugger and a server connection tool, substantially improving performance on these tasks.

Abstract:

Although language model (LM) agents have demonstrated increased performance in multiple domains, including coding and web-browsing, their success in cybersecurity has been limited. We present *EnIGMA*, an LM agent for autonomously solving Capture The Flag (CTF) challenges. We introduce new tools and interfaces to improve the agent's ability to find and exploit security vulnerabilities, focusing on interactive terminal programs. These novel *Interactive Agent Tools* enable LM agents, for the first time, to run interactive utilities, such as a debugger and a server connection tool, which are essential for solving these challenges.
Empirical analysis on 390 CTF challenges across four benchmarks demonstrate that these new tools and interfaces substantially improve our agent's performance, achieving state-of-the-art results on NYU CTF, Intercode-CTF, and CyBench. Finally, we analyze data leakage, developing new methods to quantify it and identifying a new phenomenon we term *soliloquizing*, where the model self-generates hallucinated observations without interacting with the environment.

Lay Summary:

Modern computer systems are complex, and even small mistakes in their code can open the door to hackers. To find and fix these vulnerabilities before they can be exploited, security researchers use simulated hacking challenges called "Capture The Flag" (CTF) competitions. These puzzles mimic real-world attacks and require deep technical knowledge to solve.

We created **EnIGMA**, an AI agent powered by language models, that can *solve these complex cybersecurity challenges on its own*. EnIGMA uses special tools that let it interact with computer systems like a cyber expert would—connecting to servers, analyzing and debugging code. These tools help it understand and exploit vulnerabilities in ways previous AI systems couldn’t.

Our experiments show that *EnIGMA outperforms earlier AI agents on hundreds of security tasks*. This opens new possibilities for using AI not just to assist experts, but to independently detect vulnerabilities in software—potentially speeding up security audits and reducing the risk of cyberattacks before they happen.

Application-Driven Machine Learning: This submission is on Application-Driven Machine Learning.

Link To Code: https://github.com/SWE-agent/SWE-agent/tree/v0.7

Primary Area: Applications

Keywords: Language Model Agents, Language Model, Agents, Security, Vulnerabilities, Capture The Flag

Submission Number: 4975