Modeling and reconstruction of 3D humans

Enric Corona

Abstract: English) Understanding humans in images has been a long-standing goal in Computer Vision. Recently, modelling and generation of virtual humans has become a popular area of research spurred by the success of deep learning, and motivated by the wide range of applications they would enable in AR/VR, fashion or the movie industry. However, obtaining and realistically representing avatars is a complex task, due to the complexity of body articulation and variance in appearance and clothing. Moreover, humans constantly interact with the environment, and modelling humans in interaction is necessary to fully comprehend our motion and actions, or naturally represent avatars in virtual scenes. This thesis explores the problem of 3D human modeling and reconstruction from monocular RGB images. First, we propose a method to recover human pose and shape given images or pointclouds with the goal of obtaining precise body measures. While accurate models of the body are essential to obtain reconstructions with similar characteristics, so far they lack hair, clothing, and personal details. Therefore, we next reconstruct these properties considering both full body humans or hands. We conceive methods that enable control over the 3D reconstruction, making it easy to animate the resulting avatars, perform cloth editing or human relighting, given just a monocular image. We next analyze the effect of the environment in human modelling tasks and show that contextual information enhances human motion forecasting methods. Finally, we propose a new task, dataset and method to generate realistic human-object interactions on multi-object scenes. All methods have been extensively evaluated with real data. In summary, in this thesis we propose a collection of tools for modelling and reconstruction of 3D humans, providing a step forward in the direction of creating both realistic and controllable avatars of full-body humans or human hands.