A Confidence-based Multipath Neural-symbolic Approach for Visual Question Answering

Yajie Bao, Tianwei Xing, Xun Chen

Keywords: Neural symbolic AI, Visual question answering, Confidence-based neural-symbolic methods

TL;DR: A confidence-based neural-symbolic method for efficient visual question answering.

Abstract: Neural-symbolic (NS) learning provides an efficient approach for visual question answering (VQA) by combining the advantages of neural network learning and symbolic reasoning. However, the uncertainty of the neural networks (NN) learning in the existing NS methods has not been considered, and one single answer is provided for a question without confidence evaluation. To tackle this problem, we propose a confidence-based NS (CBNS) framework to evaluate the confidence of the NN modules based on uncertainty quantification and make inferences based on the confidence evaluations. Specifically, CBNS includes a probabilistic question parser that generates multiple program candidates with confidence evaluations. CBNS also includes a probabilistic scene perception module which provides object-based scene representation and confidence evaluations for each attribute of objects in an image. The object-based scene representation and the programs with confidence evaluations are used for evaluating the confidence of answers through the inference process. The proposed framework is model-agnostic and compatible with mainstream NS VQA architectures. Experiments on CLEVR demonstrate that the proposed framework enables confidence-based reasoning for the complex VQA task and leads to a promising performance improvement with a significantly reduced computation cost.