Multi-Modal Conditional GAN: Data Synthesis in the Medical Domain

Jonathan David Ziegler, Sajanth Subramaniam, Michela Azzarito, Orla Doyle, Peter Krusche, Thibaud Coroller

Keywords: medical data, 3d images, generative adversarial networks, data synthesis, conditional synthesis, multimodal GAN, multimodal data synthesis, computed tomography image, clinical data

TL;DR: Introduction of a method capable of conditionally synthesizing correlated full-scale 3d images and structured tabular data, applied to multimodal medical datasets.

Abstract: Despite continuous collection of linked clinical and imaging datasets within the drug development process, it remains challenging to analyze those data to improve our understanding of disease and treatment. Data collection is often implemented inconsistently across studies or study sites, specific data modalities may be missing (e.g. lab measurements or medical images), and patient consent and data privacy laws constrain the purpose for which data may be used. In this paper we propose a method for conditional data generation across tabular and imaging modalities as a solution to overcome some of these challenges by generating synthetic patient data that are both realistic and complete across modalities. Our method, the multi-modal conditional GAN (MMCGAN), combines a conditional GAN for tabular data alongside a model for conditional 3D image synthesis at variable resolution. Our method brings a novel combination of capabilities: joint, scalable and efficient conditional data synthesis for clinical and full resolution 3D imaging data.