Semantic facial features and expression manipulation using multi-level IC-DGAN framework

Fakhar Abbas, Araz Taeihagh

Abstract: In recent years, the manipulation and decomposition of facial expressions and features have gained significant attention due to their extensive applications in diverse fields, including media entertainment and biometric forensics. However, leveraging these frameworks for photo enhancement encounters substantial challenges, primarily due to the adaptive nature of facial features, which are highly susceptible to alterations in illumination, pose variations, and other factors significantly affecting their appearance. This study proposes an improved clustered-based deep generative adversarial network (IC-DGAN) framework featuring parallel generators and discriminators with loss functions for semantic expression and facial feature manipulation. The key concept involves guiding the deep generative framework through clustering and semantic feature prediction, facilitating synchronized decomposition across multiple levels, and generating diverse facial images. Our framework fuses the scale-invariant feature transform (SIFT) and the K-means cluster to autonomously partition semantic features and facial expressions into distinct dimensions. Combining clustering and latent space optimization aids our framework in generating more realistic results at specific abstraction levels. Additionally, for facial feature manipulation, our framework involves pretraining the feature prediction model by reversing the synthesized facial images to the IC-DGAN latent space. Through comprehensive experiments across various extensive datasets, we evaluated the efficacy of our framework against baseline methods. Our findings demonstrate that our framework efficiently mitigates unexpected portrait diversities and expression impacts compared to baseline approaches, leading to improved manipulations and reduced distortions.