KDA: A Knowledge-Distilled Attacker for Scalable LLM Red Teaming
Keywords: Jailbreak attack; Large Language Models; Adversarial Attack; Red Teaming
TL;DR: We propose an efficient jailbreak framework that automatically generates black-box, coherent, and diverse attack prompts independent of commercial LLMs.
Abstract: Jailbreak attacks exploit specific prompts to bypass LLM safeguards and generate harmful or inappropriate content. Recently, numerous approaches have emerged for generating jailbreak attacks across diverse malicious scenarios. However, these methods often suffer from critical limitations such as the reliance on handcrafted prompts, the necessity for white-box access to target LLMs, the generation of monotonous prompts, or the dependence on expensive queries to commercial LLMs. Moreover, these methods typically require considerable time to generate jailbreak attacks. In this paper, we propose a Knowledge-Distilled Attacker (KDA) that leverages existing realistic and semantically meaningful prompts to learn a model that efficiently produces successful attacks. Specifically, we finetune an open-source LLM on a diverse set of attack prompts, enabling our framework to automatically generate black-box, coherent, and diverse attack prompts independent of commercial LLMs. Our KDA achieves a 100% success rate on multiple state-of-the-art LLMs while only requiring less than 10 seconds per attack generation. Further, using KDA, we introduce the RedTeam-10k dataset, a large-scale dataset of 10,000 harmful attack prompts inducing malicious LLM behavior spanning 12 categories such as bias, hate, and illegal activities. This dataset is 20x larger than any existing attack prompt dataset, positioning KDA as a powerful tool for large-scale adversarial testing.
Primary Area: alignment, fairness, safety, privacy, and societal considerations
Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
Submission Guidelines: I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.
Reciprocal Reviewing: I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.
Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.
No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.
Submission Number: 11757
Loading