NeMo: Neural Mesh Models of Contrastive Features for Robust 3D Pose Estimation

Angtian Wang, Adam Kortylewski, Alan Yuille

Keywords: Pose Estimation, Robust Deep Learning, Contrastive Learning, Render-and-Compare

Abstract: 3D pose estimation is a challenging but important task in computer vision. In this work, we show that standard deep learning approaches to 3D pose estimation are not robust to partial occlusion. Inspired by the robustness of generative vision models to partial occlusion, we propose to integrate deep neural networks with 3D generative representations of objects into a unified neural architecture that we term NeMo. In particular, NeMo learns a generative model of neural feature activations at each vertex on a dense 3D mesh. Using differentiable rendering we estimate the 3D object pose by minimizing the reconstruction error between NeMo and the feature representation of the target image. To avoid local optima in the reconstruction loss, we train the feature extractor to maximize the distance between the individual feature representations on the mesh using contrastive learning. Our extensive experiments on PASCAL3D+, occluded-PASCAL3D+ and ObjectNet3D show that NeMo is much more robust to partial occlusion compared to standard deep networks, while retaining competitive performance on non-occluded data. Interestingly, our experiments also show that NeMo performs reasonably well even when the mesh representation only crudely approximates the true object geometry with a cuboid, hence revealing that the detailed 3D geometry is not needed for accurate 3D pose estimation.

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics

One-sentence Summary: We introduce NeMo, a rendering-based approach to 3D pose estimation that models objects in terms of neural feature activations, instead of image intensities.

Code: [ Angtian/NeMo](https://github.com/Angtian/NeMo)

Data: [ObjectNet3D](https://paperswithcode.com/dataset/objectnet3d), [OccludedPASCAL3D+](https://paperswithcode.com/dataset/occludedpascal3d), [PASCAL3D+](https://paperswithcode.com/dataset/pascal3d-2)

Community Implementations: [ 1 code implementation](https://www.catalyzex.com/paper/arxiv:2101.12378/code)