Open-Set Synthesis for Free-Viewpoint Human Body Reenactment of Novel Poses

Zhicheng Sheng, Fan Liu, Meng Liu, Feng Zheng, Liqiang Nie

Abstract: Free-viewpoint human body reenactment aims to generate authentic and coherent poses for a source subject based on a target body pose skeleton. While current methods are proficient in reproducing existing poses, they falter in generating novel poses, often yielding blurry results. In this paper, we propose a method for open-set pose synthesis, utilizing multi-view images to generate novel poses from arbitrary viewpoints. Our method begins with building the neural radiance fields (NeRF) using multi-view images. While this NeRF is adept at rendering specific free-viewpoint refined poses, it struggles with sharp results for novel poses. To address this, we introduce a 2D novel pose diffusion (2D-NPD) module and a view-consistent NeRF optimization (VCNeRF-O) strategy. The 2D-NPD performs body reenactment in the 2D domain to generate a set of refined novel pose images. In particular, we introduce a motion adapter tailored for the stable diffusion (SD) model to generate novel poses while preserving the cloth texture. To ensure seamless motion and image clarity, we further devise a dual warp loss function for the motion adapter. Moreover, to generate fine-grained novel poses while maintaining viewpoint consistency, we develop an innovative VCNeRF-O to optimize the NeRF. Experiments demonstrate that our approach outperforms existing techniques in terms of texture quality and consistency in the open-set synthesis of novel poses.