All-in-One: Prompt-Driven Mixture of Hallucination-Aware Experts for Universal Anomaly Detection Across Multi-Modal Multi-Organ Medical Images
Keywords: universal anomaly detection, medical images, MoE, hallucinatory anomalies
Abstract: Unsupervised anomaly detection in medical images facilitates practical clinical adoption by identifying abnormalities without relying on scarce and costly annotated data. However, prior works have predominantly focused on specialized models for individual organs and modalities, impeding knowledge transfer and scalable deployment. In this paper, we investigate a task of universal anomaly detection guided by natural language prompts. We propose a prompt-driven mixture of experts framework that detects anomalies across multiple organs and modalities within a single network. Specifically, our method comprises encoders for vision and text, a routing network, and a mixture of hallucination-minimized expert decoders. An image and a prompt describing the organ and modality are fed to the encoders. The routing network then selects specialized yet collaborative expert decoders to analyze the image. We observe that anomaly detection models often erroneously identify normal image regions as anomalous, a phenomenon we term ``hallucinatory anomaly''. To address this issue, we design hallucination-aware experts that produce improved anomaly maps by jointly learning reconstruction and minimizing these false positives. For comprehensive evaluation, we curate a diverse dataset of 12,153 images spanning 5 modalities and 4 organs. Extensive experiments demonstrate state-of-the-art anomaly detection performance in this universal setting. Moreover, the natural language conditioning enables interpretability and user interaction. The code and data will be made publicly available.
Primary Area: unsupervised, self-supervised, semi-supervised, and supervised representation learning
Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
Submission Guidelines: I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.
Reciprocal Reviewing: I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.
Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.
No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.
Submission Number: 13127
Loading