InjecGuard: Benchmarking and Mitigating Over-defense in Prompt Injection Guardrail Models
Abstract: We introduce NotInject, an evaluation dataset that systematically measures over-defense across various prompt guard models. NotInject contains 339 benign samples enriched with trigger words common in prompt injection attacks, enabling fine-grained evaluation. Our results show that state-of-the-art models suffer from over-defense issues, with accuracy dropping close to random guessing levels (60%). To mitigate this, we propose NotInject, a novel prompt guard model that incorporates a new training strategy, Mitigating Over-defense for Free (MOF), which significantly reduces the bias on trigger words. InjecGuard demonstrates state-of-the-art performance on diverse benchmarks including NotInject, surpassing the existing best model by 30.8%, offering a robust and open-source solution for detecting prompt injection attacks. The code and datasets are released at https://anonymous.4open.science/r/InjecGuard-10DA.
Paper Type: Long
Research Area: Ethics, Bias, and Fairness
Research Area Keywords: prompt injection attack, overdefense, guardrail, prompt guard model, LLM safety
Contribution Types: NLP engineering experiment, Publicly available software and/or pre-trained models, Data resources
Languages Studied: English
Submission Number: 1561
Loading