Red-Teaming NSFW Image Classifiers as Text-to-Image Safeguards

ACL ARR 2026 January Submission2922 Authors

Keywords: red-teaming, text-to-image, NSFW classification, LLM, robustness, adversarial attacks

Abstract:

Not Safe for Work (NSFW) image classifiers play a critical role in safeguarding text-to-image (T2I) systems. However, a concerning phenomenon has emerged in T2I systems -- changes in text prompts that manipulate benign image elements can result in failed detection by NSFW classifiers -- dubbed "*context shifts*." For instance, while a NSFW image of "*a nude person in an empty scene*" can be easily blocked by most NSFW classifiers, a stealthier one that depicts "*a nude person blending in a group of dressed people*" may evade detection. We ask: how to systematically reveal NSFW image classifiers' failure against such context shifts?

Towards this end, we present an automated red-teaming framework that leverages a set of generative AI tools. We propose an **exploration-exploitation** approach: **First**, in the *exploration* stage, we synthesize a diverse and massive 36K NSFW image dataset that facilitates our study of context shifts. We find that varying fractions (e.g., 4.1% to 36% nude and sexual content) of the dataset are misclassified by NSFW image classifiers like GPT-4o and Gemini. **Second**, in the *exploitation* stage, we leverage these failure cases to train a specialized LLM that rewrites unseen seed prompts into more evasive versions, increasing the likelihood of detection evasion by up to 6 times. Alarmingly, we show **these failures translate to real-world T2I and even T2V systems** like DALL-E 3, Sora, Nano Banana, and Veo 3 -- beyond the open-weight image generators in our main study. For example, querying DALL-E 3 with prompts rewritten by our approach increases the chance of obtaining NSFW images from 0 to over 50%.

Paper Type: Long

Research Area: NLP Applications

Research Area Keywords: security/privacy, cross-modal content generation, safety and alignment

Languages Studied: English

Submission Number: 2922