SIVA: Self-Improving Vulnerability Agent

Valentin Walischewski, Giulio Zizzo, Kevin N. Webster

Keywords: LLM, Agents, Cybersecurity, Code Vulnerabilities, Prompt Optimisation, Meta-Learning

TL;DR: The paper presents SIVA, a LLM-based agent, that use memory guided meta-learning to self-improve at code vulnerability detection, achieving state-of-the-art performance.

Abstract: In the ever more digitalized world of today, code vulnerabilities pose a critical threat to our privacy, economy, safety, and infrastructure. Existing automated code vulnerability detection methods suffer from high false positive rates, poor generalization and their inability to adapt to changing vulnerability landscapes. To address these challenges we propose SIVA, a self-improving LLM-based vulnerability detection agent, using memory-guided meta-learning for dynamic prompt optimization. SIVA showed strong learning capabilities, improving its F1 score from 58\% to 95\% in $5$ iterations, significantly outperforming previous state-of-the-art multi-agent systems ($\approx 53\%$ F1) on real-life vulnerability datasets. Furthermore, SIVA generalized well across 7 programming languages (93\% F1), successfully transferring learned vulnerability concepts between them.

Submission Number: 78