Uncertainty-Aware Symbolic State Monitoring with Vision-Language Models and Gaussian Naive Bayes

Lioba Schürmann, Kento Kawaharazuka, Freek Stulp, Samuel Bustamante

Keywords: vision-language models, uncertainty estimation, service robotics

TL;DR: We propose a Naive Bayes classifier trained on a small scene-agnostic data set of reduced vision-language embeddings for estimating the symbolic state of novel scenes with uncertainty quantification.

Abstract: Autonomous robots require generalizable reasoning abilities to interact intelligently with unknown environments, including the ability to estimate the symbolic state of their surrounding objects. Vision-language models (VLMs) have shown promising advances towards novel scene understanding due to their strong generalization capabilities. Contrastive VLMs encode the semantics of vision embeddings by learning to align them with textual embeddings. These models, however, lack an uncertainty quantification of the resulting alignment, leading to unreliable results when monitoring an environment’s symbolic state. To address this limitation, we propose a method that integrates vision-language embeddings into a Bayesian framework for classification. Our approach constructs a representational subspace of the embeddings that captures the symbolic states to be monitored and learns Gaussian distributions on a small set of scene-agnostic support data. We model the symbolic state estimation as a classification problem with a Naive Bayes classifier, which provides information on the classification confidence and enables the integration of informative priors into the estimation process. We demonstrate the effectiveness of our method through two proof-of-concept experiments utilizing real-world data from kitchen environments.

Email Sharing: We authorize the sharing of all author emails with Program Chairs.

Data Release: We authorize the release of our submission and author names to the public in the event of acceptance.

Submission Number: 43