Go to ICLR 2025 Conference homepageLeveraging System-Prompt Attention to Counteract Novel Jailbreak Attacks
In the past few years, Language Models (LMs) have shown par-human capabilities in several domains. Despite their practical applications and exceeding user consumption, they are susceptible to jailbreaks when malicious inputs exploit the LM's weaknesses, causing it to deviate from its intended behavior. Current defensive strategies either classify the input prompt as adversarial or prevent LMs from generating harmful outputs. The primary challenge is that the current defense techniques are built against known and established jailbreaking patterns while work poorly against novel attacks. In this research, we propose an end-to-end framework for generating novel attack patterns and demonstrate how the proposed defense approach can generalize over known and unknown attack patterns. Attack patterns are generated using a novel self-learning large language model (LLM)-based multi-agent system with closed loop feedback called ALMAS, which stands for Attack using LLM-based Multi-Agent Systems. We demonstrate that system-prompt attention from Small Language Models (SLMs) can be used to characterize adversarial prompts providing a novel explainable and cheaper defense approach called AttentionDefense. The proposed AttentionDefense is evaluated against existing jailbreak benchmark datasets as well as the novel jailbreaks generated using ALMAS. Ablation studies demonstrate that SLM-based AttentionDefense has equivalent or better jailbreak detection performance as compared to text embedding based classifiers and GPT-4 zero-shot detectors. Our research suggests that the attention mechanism is an integral component in understanding and explaining how LMs respond to malicious inputs that is not captured in the semantic meaning of text embeddings. Additionally, for practical purposes AttentionDefense is an ideal solution as it has the computation requirements of a small LM but the performance of a LLM detector.