Face Photo-Sketch Recognition Using Bidirectional Collaborative Synthesis Network

Seho Bae, Nizam Ud Din, Hyunkyu Park, Juneho Yi

Abstract: This research features a deep learning based framework to address the problem of matching a given face sketch image against a face photo database. The photo-sketch matching problem is challenging because 1) modality gap between photo and sketch is very large, and 2) the number of paired photo/ sketch data is insufficient to train deep network. To circumvent the problem of large modality gap, our approach is to use an intermediate latent space between the two modalities. We effectively align the distributions of the two modalities in this latent space by employing a bidirectional (photo → sketch and sketch → photo) collaborative synthesis network. A StyleGAN-like architecture is utilized to make the intermediate latent space be equipped with rich representation power. To resolve the problem of insufficient training samples, we introduce a three-step training scheme. Extensive evaluation on public composite face sketch database confirms superior performance of our method compared to existing state-of-the-art methods. The proposed methodology can be employed in matching other modality pairs.