Learning Face Image Super-Resolution Through Facial Semantic Attribute Transformation and Self-Attentive Structure Enhancement

Mengyan Li, Zhaoyu Zhang, Jun Yu, Chang Wen Chen

Abstract: Face super-resolution is a domain-specific super-resolution (SR) problem of generating high-resolution (HR) face images from low-resolution (LR) inputs. Even though existing face SR methods have achieved great performance on the global region evaluation, most of them cannot restore local attributes and structure reasonably, especially to ultra-resolve tiny LR face images (16 × 16 pixels) to its larger version (8 × upscaling factor). In this paper, we propose an open source face SR framework based on facial semantic attribute transformation and self-attentive structure enhancement. Specifically, the proposed framework introduces face semantic information (i.e., face attributes) and face structure information (i.e., face boundaries) in a successive two-stage fashion. In the first stage, an Attribute Transformation Network (AT-Net) is established. It upsamples LR face images to HR feature maps and then combines facial attributes with these features to generate the intermediate HR results with rational attributes. In the second stage, a Structure Enhancement Network (SE-Net) is built. It simultaneously extracts face features and estimates facial boundary heatmaps from the inputs, and then fuses them to output the final HR face images. Extensive experiments demonstrate that our method achieves superior super-resolved results and outperforms the state-of-the-art methods.