Unsupervised Disentanglement of Pose, Appearance and Background from Images and Videos

Aysegul Dundar, Kevin J. Shih, Animesh Garg, Robert Pottorff, Andrew Tao, Bryan Catanzaro

Abstract:

Unsupervised landmark learning is the task of learning semantic keypoint-like representations without the use of expensive
input keypoint annotations. A popular approach is to factorize an image into a pose and appearance data stream, then to reconstruct
the image from the factorized components. The pose representation should capture a set of consistent and tightly localized landmarks
in order to facilitate reconstruction of the input image. Ultimately, we wish for our learned landmarks to focus on the foreground object of
interest. However, the reconstruction task of the entire image forces the model to allocate landmarks to model the background. Using a
motion-based foreground assumption, this work explores the effects of factorizing the reconstruction task into separate foreground and
background reconstructions in an unsupervised way, allowing the model to condition only the foreground reconstruction on the
unsupervised landmarks. Our experiments demonstrate that the proposed factorization results in landmarks that are focused on the
foreground object of interest when measured against ground-truth foreground masks. Furthermore, the rendered background quality is
also improved as ill-suited landmarks are no longer forced to model this content. We demonstrate this improvement via improved image
fidelity in a video-prediction task. Code is available at https://github.com/NVIDIA/UnsupervisedLandmarkLearning