SoK: Large Language Models in Security Code Review and Testing

JSYS 2025 October Papers Submission3 Authors

Keywords: LLMs, software security, security testing, vulnerability detection, fuzz testing

TL;DR: This is a SoK paper presenting and discussing practical applications of LLMs in software security, specifically in code vulnerability detection, fuzz testing and exploit generation.

Abstract: In this paper, we present and discuss practical applications of Large Language Models (LLMs) in software security, concretely in code vulnerability detection, fuzz testing and exploit generation. Measurements of various research outcomes are analysed to answer questions about the performance of LLM in those fields, including a comparison with tools following traditional approaches. In addition, the drawbacks and a future overlook with a delineation of technical challenges are given. Challenges are found in the cost- and time-intensive training of LLM, the limited context-length understanding of program code, the high false positive rate because of hallucinations, and keeping the data up-to-date so that definitions of newly detected vulnerabilities are contained.

Area: System Security

Type: Systemization of Knowledge (SoK)

Revision: Yes

Previous Version: https://openreview.net/forum?id=hMkoe4C44D

Submission Number: 3