You've Got to Feel It to Believe It: Multi-Modal Bayesian Inference for Semantic and Property Prediction

Parker Ewen, Hao Chen, Yuzhen Chen, Anran Li, Anup Bagali, Gitesh Gunjal, Ram Vasudevan

Abstract:

Robots must be able to understand their surround-
ings to perform complex tasks in challenging environments and
many of these complex tasks require estimates of physical prop-
erties such as friction or weight. Estimating such properties using
learning is challenging due to the large amounts of labelled data
required for training and the difficulty of updating these learned
models online at run time. To overcome these challenges, this
paper introduces a novel, multi-modal approach for representing
semantic predictions and physical property estimates jointly in
a probabilistic manner. By using conjugate pairs, the proposed
method enables closed-form Bayesian updates given visual and
tactile measurements without requiring additional training data.
The efficacy of the proposed algorithm is demonstrated through
several simulation and hardware experiments. In particular, this
paper illustrates that by conditioning semantic classifications on
physical properties, the proposed method quantitatively outper-
forms state-of-the-art semantic classification methods that rely
on vision alone. To further illustrate its utility, the proposed
method is used in several applications including to represent
affordance-based properties probabilistically and a challenging
terrain traversal task using a legged robot. In the latter task,
the proposed method represents the coefficient of friction of the
terrain probabilistically, which enables the use of an on-line risk-
aware planner that switches the legged robot from a dynamic gait
to a static, stable gait when the expected value of the coefficient
of friction falls below a given threshold. Videos of these case
studies as well as the open-source C++ and ROS interface can
be found at https://roahmlab.github.io/multimodal mapping/