Go to ICLR 2025 Conference homepageSafeVision: Efficient Image Guardrail with Robust Policy Adherence and Explainability
As image generation models become increasingly prevalent, the need for efficient and transparent guardrails against unsafe content is more critical than ever. Traditional unsafe image classifiers, limited to predefined categories, often misclassify content due to the pure feature-based learning rather than semantic-based reasoning and struggle to adapt to emerging threats. The time and resources required for retraining on new harmful categories further hinder their ability to respond to evolving threats. To address these challenges, we propose SafeVision, a novel image guardrail system that integrates human-like understanding and reasoning with scalability. Within SafeVision, we propose an effective data collection and generation, policy-following training pipeline, and a customized loss function. In particular, we propose an efficient diverse QA generation and training strategy to enhance the effectiveness of the training process. SafeVision is able to follow given safety policies during inference time to guardrail against new risk categories and thus avoid expensive retraining, provide accurate risky content predictions, and provide precise explanations. SafeVision operates in two modes: 1) rapid classification mode, and 2) comprehension mode that provides both classification and human-readable explanations. In addition, considering the limitations of existing unsafe image benchmarks, which contain either only binary or limited categories, we provide VisionHARM-500K, a high-quality unsafe image benchmark comprising over 500k images to cover a wide array of risky categories. This dataset significantly broadens the scope and depth of unsafe image benchmarks. Through comprehensive experiments, we show that SafeVision achieves state-of-the-art performance in both efficiency and accuracy, with an accuracy of 91.77% on the VisionHARM-500K test set (17.77% higher than GPT-4O) and an inference time of 0.0979 seconds per image (over 50 times faster than GPT-4O). SafeVision sets a new standard for comprehensive, policy-following, and explainable image guardrail models, delivering state-of-the-art performance while aligning with human reasoning and enabling scalable adaptation to emerging threats.