Dual-function discriminator for semantic image synthesis in variational GANs

Aihua Ke, Bo Cai, Yujie Huang, Jian Luo, Yaoxiang Yu, Le Li

Abstract: Semantic image synthesis aims to generate target images conditioned on given semantic labels, but existing methods often struggle with maintaining high visual quality and accurate semantic alignment. To address these challenges, we propose VD-GAN, a novel framework that integrates advanced architectural and functional innovations. Our variational generator, built on an enhanced U-Net architecture combining a pre-trained Swin transformer and CNN, captures both global and local semantic features, generating high-quality images. To further boost performance, we design two innovative modules: the Conditional Residual Attention Module (CRAM) for dimensionality reduction modulation and the Channel and Spatial Attention Mechanism (CSAM) for extracting key semantic relationships across channel and spatial dimensions. Additionally, we introduce a dual-function discriminator that not only distinguishes real and synthesized images, but also performs multi-class segmentation on synthesized images, guided by a redefined class-balanced cross-entropy loss to ensure semantic consistency. Extensive experiments show that VD-GAN outperforms the latest supervised methods, with improvements of (FID, mIoU, Acc) by (5.40\%, 4.37\%, 1.48\%) and increases in auxiliary metrics (LPIPS, TOPIQ) by (2.45\%, 23.52\%). The code will be available at \texttt{https://github.com/ah-ke/VD-GAN.git}.