Hybrid Neuro-Symbolic Reasoning based on Multimodal Fusion

Subrata Das, Bodong Zhou

Keywords: Neural Networks, Deep Learning, Symbolic Reasoning, Multimodal Fusion, Word Embedding, Rule-based Reasoning

TL;DR: A hybrid neural/symbolic modeling to enhance complex image classifications using commonsense knowledge.

Abstract:

Deep neural models and symbolic Artificial Intelligence (AI) systems have contrasting advantages and disadvantages. Neural models can be trained from raw, incomplete and noisy data to obtain abstraction of features at various levels, but their uninterpretability is well-known. On the other hand, the traditional rule-based symbolic reasoning encodes domain knowledge, but its failure is often attributed to the acquisition bottleneck. We propose to build a hybrid learning and reasoning system which is based on multimodal fusion approach that brings together
advantageous features from both the paradigms. Specifically, we enhance convolutional neural networks (CNNs) with the structured information of ‘if-then’ symbolic logic rules obtained via word embeddings corresponding to propositional symbols and terms. With many dozens of intuitive rules relating the type of a scene with its typical constituent objects, we are able to achieve significant improvement over the base CNN-based classification. Our approach is extendible to handle first-order logical syntax for rules and other deep learning models.

Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.

No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics

Submission Guidelines: Yes

Please Choose The Closest Area That Your Submission Falls Into: General Machine Learning (ie none of the above)