PRISM: High-Resolution & Precise Counterfactual Medical Image Generation using Language-guided Stable Diffusion

Amar Kumar, Anita Kriz, Mohammad Havaei, Tal Arbel

Keywords: Counterfactual Image Synthesis, Diffusion, Foundation Models, Generative Models, Image Synthesis, Large Language Models

Abstract: Developing reliable and generalizable deep learning systems for medical imaging faces significant obstacles due to spurious correlations, data imbalances, and limited text annotations in datasets. Addressing these challenges requires architectures robust to the unique complexities posed by medical imaging data. The rapid advancements in vision-language foundation models within the natural image domain prompt the question of how they can be adapted for medical imaging tasks. In this work, we present PRISM, a framework that leverages foundation models to generate high-resolution, language-guided medical image counterfactuals using Stable Diffusion. Our approach demonstrates unprecedented precision in selectively modifying spurious correlations (the medical devices) and disease features, enabling the removal and addition of specific attributes while preserving other image characteristics. Through extensive evaluation, we show how PRISM advances counterfactual generation and enables the development of more robust downstream classifiers for clinically deployable solutions. To facilitate broader adoption and research, we make our code publicly available at https://github.com/Amarkr1/PRISM.

Primary Subject Area: Foundation Models

Secondary Subject Area: Generative Models

Paper Type: Both

Registration Requirement: Yes

Reproducibility: https://github.com/Amarkr1/PRISM

Visa & Travel: Yes

Submission Number: 230