An evidence-based neuro-symbolic framework for ambiguous image scene classification

Giulia Murtas, Veselka Boeva, Elena Tsiporkova

Keywords: Computer Vision, Image Scene Classification, Modal Logic, Evidence Theory, Multi- valued Mapping

TL;DR: Neuro-symbolic approach integrating deep learning, modal logic and evidence theory to tackle ambiguous scene classification, applied to the use case of abandoned object detection.

Track: Main Track

Abstract: In this study, we propose a novel neuro-symbolic approach to deal with the inherent ambiguity in image scene classification, combining the usage of pre-trained deep learning (DL) models with concepts from modal logic and evidence theory. The DL models are used to detect objects and estimate their depth in a set of labeled images. The obtained outputs are employed to form a dataset of instances characterizing the possible classes. Subsequently, a multi-valued mapping is defined between the data instances and the considered images resulting into each image being represented by the set of instances associated with it. The obtained mapping is utilized to infer necessity and possibility conditions of each class, or equivalently its upper (plausibility) and lower (belief) probabilities. Based on these interval evaluations, a rule-based and a score-based classifiers are built. The overall method is explainable and directly interpretable, robust to data scarcity and data imbalance. The presented framework is studied and evaluated on an abandoned bag detection use case.

Paper Type: Long Paper

Submission Number: 33