Hybrid Diffusion Model for Breast Ultrasound Image Augmentation

Farhan Fuad Abir, Sanjeda Sara Jennifer, Niloofar Yousefi, Laura Brattain

Submission Track: Short papers presenting ongoing research or work submitted to other venues (up to 5 pages, excluding references)

Keywords: diffusion model, breast ultrasound, medical image augmentation, textual inversion

TL;DR: Hybrid diffusion model with img-to-img refinement improves the fidelity of synthetic breast ultrasound images.

Abstract: We propose a hybrid diffusion-based augmentation framework to overcome the critical challenge of limited and imbalanced data in breast ultrasound (BUS) datasets. Unlike conventional augmentations, our approach captures ultrasound-specific features such as speckle noise by combining text-to-image generation with image-to-image (img2img) refinement as well as fine-tuning using LoRA and textual inversion (TI). Our method generated realistic, class-consistent images on an open-source Kaggle breast ultrasound image dataset (BUSI). Incorporating TI and img2img refinement on the Stable Diffusion v1.5 backbone reduced the Fréchet Inception Distance (FID) from 45.97 to 33.29, demonstrating a substantial gain in fidelity while maintaining comparable downstream classification performance. Overall, the proposed framework effectively mitigates the low-fidelity limitations of synthetic ultrasound images and enhances the quality of augmentation for robust diagnostic modeling.

Submission Number: 59