Iterative Training of Language Models with Opponent Modeling for Red Teaming Data Generation
Keywords: LLM Safety, Red Teaming of LLMs, Synthetic Data Generation
Abstract: Large language models (LLMs) exhibit impressive capabilities across various tasks but are also prone to generating harmful outputs. To address this risk, we explore an iterative red teaming approach that focuses on adversarial prompt refinement. Although this method improves attack success rates, it faces challenges of slow progress, high computational cost, and limited prompt diversity. To overcome these limitations, we propose a training framework using a smaller model, Llama3.1-8B, integrated with opponent modeling to simulate responses and enhance attack performance. Our method achieves a 74.95% attack success rate on Llama2-7b-Chat and 69.10% on Llama3-8b-Instruct, while also preserving prompt diversity. Our analysis of the trained red teaming LLM reveals that red teaming abilities are densely embedded in model parameters, unlike the sparsity observed in safety alignment features. We release the data and code to facilitate further research on improving LLM safety alignment.
Primary Area: alignment, fairness, safety, privacy, and societal considerations
Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
Submission Guidelines: I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.
Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.
No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.
Submission Number: 2630
Loading