S2TD-Face: Reconstruct a Detailed 3D Face with Controllable Texture from a Single Sketch

Zidu Wang, Xiangyu Zhu, Jiang Yu, Tianshuo Zhang, Zhen Lei

Abstract: 3D textured face reconstruction from sketches applicable in many scenarios such as animation, 3D avatars, artistic design, missing people search, etc., is a highly promising but underdeveloped research topic. On the one hand, the stylistic diversity of sketches leads to existing sketch-to-3D-face methods only being able to handle pose-limited and realistically shaded sketches. On the other hand, texture plays a vital role in representing facial appearance, yet sketches lack this information, necessitating additional texture control in the reconstruction process. This paper proposes a novel method for reconstructing controllable textured and detailed 3D faces from sketches, named S2TD-Face. S2TD-Face introduces a two-stage geometry reconstruction framework that directly reconstructs detailed geometry from the input sketch. To keep geometry consistent with the delicate strokes of the sketch, we propose a novel sketch-to-geometry loss that ensures the reconstruction accurately fits the input features like dimples and wrinkles. Our training strategies do not rely on hard-to-obtain 3D face scanning data or labor-intensive hand-drawn sketches. Furthermore, S2TD-Face introduces a texture control module utilizing text prompts to select the most suitable textures from a library and seamlessly integrate them into the geometry, resulting in a 3D detailed face with controllable texture. S2TD-Face surpasses existing state-of-the-art methods in extensive quantitative and qualitative experiments. The code will be publicly available.

Primary Subject Area: [Experience] Multimedia Applications

Secondary Subject Area: [Experience] Multimedia Applications, [Content] Media Interpretation

Relevance To Conference: This paper focuses on the sketch-to-3D-face task and introduces a method called S2TD-Face, which utilizes a face sketch and a text prompt for texture description as input, capable of reconstructing topology-consistent 3D faces with fine-grained geometry that precisely matches the input sketch. It allows users to control the texture of the reconstruction through text prompts. To ensure the framework accurately reflects the delicate features of the input, we propose a novel sketch-to-geometry loss to supervise both coarse and detailed geometry. This work applies broadly to various scenarios, including animation, 3D avatars, artistic design, criminal investigation, etc.

Submission Number: 2815