$\alpha$-MDF: An Attention-based Multimodal Differentiable Filter for Robot State Estimation

Xiao Liu, Yifan Zhou, Shuhei Ikemoto, Heni Ben Amor

Keywords: Differentiable Filters, Sensor Fusion, Multimodal Learning

TL;DR: This paper introduces an attention-based multimodal differentiable filter framework, which learns multimodal latent representations from data and achieves notable advancements in robot state estimation for both rigid body and soft robot dynamics.

Abstract: Differentiable Filters are recursive Bayesian estimators that derive the state transition and measurement models from data alone. Their data-driven nature eschews the need for explicit analytical models, while remaining algorithmic components of the filtering process intact. As a result, the gain mechanism -- a critical component of the filtering process -- remains non-differentiable and cannot be adjusted to the specific nature of the task or context. In this paper, we propose an attention-based Multimodal Differentiable Filter ($\alpha$-MDF) which utilizes modern attention mechanisms to learn multimodal latent representations. Unlike previous differentiable filter frameworks, $\alpha$-MDF substitutes the traditional gain, e.g., the Kalman gain, with a neural attention mechanism. The approach generates specialized, context-dependent gains that can effectively combine multiple input modalities and observed variables. We validate $\alpha$-MDF on a diverse set of robot state estimation tasks in real world and simulation. Our results show $\alpha$-MDF achieves significant reductions in state estimation errors, demonstrating nearly 4-fold improvements compared to state-of-the-art sensor fusion strategies for rigid body robots. Additionally, the $\alpha$-MDF consistently outperforms differentiable filter baselines by up to 45% in soft robotics tasks. The project is available at alpha-mdf.github.io and the codebase is at github.com/ir-lab/alpha-MDF

Student First Author: yes

Instructions: I have read the instructions for authors (https://corl2023.org/instructions-for-authors/)

Code: https://github.com/ir-lab/alpha-MDF

Video: https://www.youtube.com/watch?v=OdBMquRUTdU

Website: https://alpha-mdf.github.io/