FuzzyCLIP: Clustering-Driven Stacked Prompt in Zero-Shot Anomaly Detection

Yanning Hou, Ke Xu, Yanran Ruan, Junfa Li, Shanshan Wang, Jianfeng Qiu

Keywords: Anomaly detection, Zero-shot anomaly detection, CLIP, Industrial defect inspection

TL;DR: We propose a novel approach, namely FuzzyCLIP, to adapt CLIP for accurate zero-shot anomaly detection.

Abstract: How to enhance the alignment of text and image features in CLIP model is a key challenge in zero-shot industrial anomaly detection tasks. Recent studies mostly rely on precise category prompts for pre-training, but this approach is prone to overfitting, which limits the generalization ability of the mode. To address this issue, we propose the concept of fuzzy prompts and introduce Clustering-Driven Stacked Prompts (CSP) along with the Ensemble Feature Alignment (EFA) module to improve the alignment between text and image features. This design significantly outperforms other methods in terms of training speed, stability, and final convergence results, showing remarkable efficiency in enhancing anomaly detection segmentation performance. What is even more surprising is that fuzzy stacked prompts exhibit strong generalization in classification tasks, enabling them to adapt to various anomaly classification tasks without any additional operations. Therefore, we further propose the Regulating Prompt Learning (RPL) module, which leverages the strong generalization ability of fuzzy stacked prompts to regularize prompt learning, thereby improving performance in anomaly detection classification tasks. We conducted extensive experiments on seven industrial anomaly detection datasets, which demonstrate that our method achieves state-of-the-art performance in zero-shot anomaly detection and segmentation tasks.

Primary Area: applications to computer vision, audio, language, and other modalities

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.

Submission Guidelines: I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.

Reciprocal Reviewing: I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.

Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.

No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.

Submission Number: 6276