What Features in Prompts Jailbreak LLMs? Investigating the Mechanisms Behind Attacks

Nathalie Maria Kirch, Severin Field, Stephen Casper

Keywords: Jailbreaks, Red-Teaming, Nonlinear probes, Linear probes, Adversarial perturbations, Latent Space Attacks, Jailbreak success, Safety and reliability, Mechanistic jailbreaks, Representation Engineering, Transferability of jailbreaks, Nonlinear features in attacks, Gemma-7B-IT

TL;DR: We compare linear and nonlinear probes to analyze latent features in jailbreak prompts, showing that successful jailbreaks in LLMs are driven by distinct nonlinear features, which can be exploited to design more effective adversarial perturbations.

Abstract: While ‘jailbreaks’ have been central to research on the safety and reliability of LLMs (large language models), the underlying mechanisms behind these attacks are not well understood. Some prior works have used linear methods to analyze jailbreak prompts or model refusal. Here, however, we compare linear and nonlinear probes to study the features in prompts that contribute to successful jailbreaks. We do this by probing for jailbreak success based only on the portions of the latent representations corresponding to prompt tokens. We first introduce a dataset of 10,800 jailbreak attempts from 35 attack methods. We then show that different jailbreaking methods work via different nonlinear features in prompts. Specifically, we find that while probes can distinguish between successful and unsuccessful jailbreaking prompts with a high degree of accuracy, they nonetheless transfer poorly to held-out attack methods. We also show that nonlinear probes can be used to mechanistically jailbreak the LLM by guiding the design of adversarial latent perturbations. These mechanistic jailbreaks are able to jailbreak Gemma-7B-IT more reliably than 34 of the 35 techniques that it was trained on. Ultimately, our results suggest that jailbreaks cannot be thoroughly understood in terms of universal or linear prompt features.

Serve As Reviewer: [e-mail], [e-mail]

Submission Number: 50