MedViTGAN: End-to-End Conditional GAN for Histopathology Image Augmentation with Vision Transformers

Meng Li, Chaoyi Li, Brian C. Lovell

Abstract:

Deep learning networks have demonstrated competitive performance for various tasks on medical images. However, obtaining promising results requires a large amount of annotated data for supervised training, which is labor-intensive.
Recently, the increasing interest in transformers has suggested their robust performance on computer vision tasks, including generative adversarial networks (GANs). In this paper, we propose a conditional GAN built on pure transformer-based architectures, named MedViTGAN, to assist in generating synthetic histopathology images for data augmentation in an end-to-end manner.
The presented model adopts a conditioned training strategy by incorporating a transformer-based auxiliary classifier to facilitate the discriminative image generation process. We further introduce an adaptive hybrid loss weighting mechanism to balance multiple losses over sources and classes to stabilize the training. Extensive experiments on the histopathology datasets show that leveraging MedViTGAN generated images results in a significant and consistent improvement in classification performance.