Performance Evaluation of Generative High-Fidelity One Shot Transfer for Virtual Kimono Try-on with Identity Loss Variation and Proposal for 3D Face Mesh Shape Loss

Takaki Fushimi, Yusuke Kameda

Abstract: This study aims to realize the transfer of face shape while preserving the hairstyle through the improvement of the Generative High-fidelity One Shot Transfer (GHOST) of the face swapping model, and to apply it to the virtual try-on for kimono. This “face shape” refers to the contour and skeleton of the entire face, as well as specific features such as the forehead, cheekbones, nose, and chin. However, transferring facial shape is a challenge for face swapping models. Therefore, we considered that the shape of the SOURCE face could be retained by increasing the weight of Identity Loss, which is responsible for reproducing the features of the source face. In this study, we tested the effect on face shape retention by adjusting the weights of Identity Loss. As a result, there was no change in facial shape, but the lighting and brightness of the face changed, and naturalness was lost. These results suggest that it is difficult to accurately transfer the face shape only by adjusting Identity Loss. To address these issues, this study proposes the introduction of Shape Loss using 3D meshes. This approach is expected to be able to accurately transfer facial shapes and reproduce lighting and brightness naturally.