SoK: Large Language Models in Security Code Review and Testing

JSYS 2025 October Papers Submission3 Authors

Keywords: LLMs, software security, security testing, vulnerability detection, fuzz testing

TL;DR: This is a SoK paper presenting and discussing practical applications of LLMs in software security, specifically in code vulnerability detection, fuzz testing and exploit generation.

Abstract: In this paper, we systematically present and discuss practical applications of Large Language Models (LLMs) in software security, concretely in code vulnerability detection, fuzz testing, and exploit generation. Measurements of various research outcomes are analyzed to answer questions about the performance of LLMs in those fields, including a comparison with tools that follow traditional approaches. In addition, the drawbacks and a future outlook, along with a delineation of technical challenges, are provided. Challenges include the cost- and time-intensive training of LLMs, the limited context-length understanding of program code, the high false positive rate due to hallucinations, and keeping the data up-to-date so that definitions of newly detected vulnerabilities are covered.

Area: System Security

Type: Systemization of Knowledge (SoK)

Revision: Yes

Previous Version: https://openreview.net/forum?id=P67TGZSttt

Submission Number: 3