Using Specularities to Boost Non-Rigid Structure-from-Motion

Agniva Sengupta, Adrien Bartoli, Karim Makki

Abstract:

Non-Rigid Structure-from-Motion (NRSfM) reconstructs the time-varying 3D shape of a deforming object
from 2D point correspondences in monocular images. Despite
promising use-cases such as the grasping of deformable objects
and visual navigation in a non-rigid environment, NRSfM
has had limited applications in robotics due to a lack of
accuracy. To remedy this, we propose a new method which
boosts the accuracy of NRSfM using sparse surface normals.
Surface normal information is available from many sources,
including structured lighting, homography decomposition of
infinitesimal planes and shape priors. However, these sources
are not always available. We thus propose a widely available
new source of surface normals: the specularities. Our first
technical contribution is a method which detects specular
highlights and reconstructs the surface normals from it. It
assumes that the light source is approximately localised, which
is widely applicable in robotics applications such as endoscopy.
Our second technical contribution is an NRSfM method which
exploits a sparse surface normal set. For that, we propose
a novel convex formulation and a globally optimal solution
method. Experiments on photo-realistic synthetic data and real
household and medical data show that the proposed method
outperforms existing NRSfM methods.