Hybrid Diffusion Model for Ultrasound Image Augmentation

Farhan Fuad Abir, Sanjeda Sara Jennifer, Niloofar Yousefi, Laura Brattain

Submission Track: Short papers presenting ongoing research or work submitted to other venues (up to 5 pages, excluding references)

Keywords: diffusion model, breast ultrasound, medical image augmentation, textual inversion

TL;DR: Hybrid diffusion model with img-to-img refinement improves the fidelity of medical ultrasound images.

Abstract: We propose a hybrid diffusion-based augmentation framework to overcome the critical challenge of limited and imbalanced data in medical ultrasound AI. Unlike conventional augmentations, our approach captures ultrasound-specific features such as speckle noise by combining text-to-image generation with image-to-image (img2img) refinement and fine-tuning using LoRA and textual inversion (TI). For the Breast Ultrasound (BUSI) dataset, our method generated realistic, class-consistent images that improved classification accuracy (90.4\% $\rightarrow$ 91.7\%), F1-score (88.7\% $\rightarrow$ 90.4\%), and achieved an AUC of 0.985. Incorporating img2img refinement further reduced the Fréchet Inception Distance (FID) to 33.29, enhancing visual fidelity without sacrificing performance. These results demonstrate that hybrid diffusion augmentation produces high-fidelity ultrasound images and strengthens downstream model reliability, offering a scalable solution to one of the most persistent barriers in clinical imaging AI.

Submission Number: 59