The use of neuro-symbolic methods to supplement the performance of deep learning based natural language inference models has witnessed a resurgence. 
In this work, we review three sets of recent results in human cognition experiments -- in natural language comprehension, in natural language inference, and in computer program comprehension - a field bearing similarities to natural language.
In light of these three works, we discuss the broader role cognitive neuroscience can play in informing the design of neuro-symbolic inference model architectures for language.

Can Cognitive Neuroscience inform Neuro-Symbolic Inference Models?

As an answer to recent contributions about the conjectured impossibility of learning meaning from surface form alone,and the dangers of large language models, we argue in this paper that an explicit distinction should be made between (i) perception, (ii) memory, and (iii) inference.  We envision a triad of interacting subsystems with corresponding responsibilities.  Perception provides the interface between  the  system  and  its  environment,  and  is  typically  realised as a language model.  Explicit Memory is a structure of concepts and relations between the concepts, in other words, a knowledge base of facts.   Inference,  finally,  corresponds to mathematical or rule-based reasoning and provides,  for ex-ample, classical logic and arithmetic. We note here that such axiomatic systems cannot be deduced from data,  only conjectured and tried against data.  Our position is that natural language systems should thus combine continuously updated language and other perception models (e.g., computer vision)with one or more symbolic knowledge bases that relieve the models from learning concepts and relations, and finally one or more inference engines to provide formal reasoning.

Perception, Memory, and Inference: The Trinity of Machine Learning

Pre-trained language models (PTLMs), such as
BERT, ELMO and XLNET, have yielded state-of-the-art performance on many natural language processing tasks. In this paper, we argue that, despite their popularity and their contextual nature, PTLMs are unable to correctly capture high-level semantics such as linking age to the date of birth and rich to net worth. 
These kind of semantically-based inferences are performed systematically by humans, which is why we would we assume that PTLMs with all their language capabilities are able to make similar predictions. We show in this position paper that PTLMs are really good at making predictions for taxonomic relationships, but fail at attribute-value semantics like in rich and net worth. 

How can BERT Understand High-level Semantics?

Emerging pre-trained language models (PTLMs), such as BERT and RoBERTa, have already achieved great success on many natural language understanding (NLU) tasks, spurring widespread interest for their potential in scientific and social areas, with accompanying criticism on their ambiguousness of reasoning, especially multi-hop cases.  Concretely, many studies have pointed out that these models lack true understandings of the reasoning process. In this work, we focus on multi-hop reasoning processes of PTLMs and perform an analysis on a logical reasoning dataset, Soft Reasoner. We first extend the dataset by constructing the implicit intermediate results during multi-hop reasoning in a semi-automatic way. Surprisingly, when testing on the extended dataset, PTLMs can even predict the correct conclusion when they cannot judge the corresponding intermediate results. To further analyze this phenomenon, we further compare PTLMs' reasoning processes with Bayesian inference processes to simulate humans' reasoning procedure.  Results show that if a model is more in line with the Bayesian process, it tends to have a better generalization ability. Our Bayesian process method can be used as a method to evaluate the generalization ability of models.

Exploring Multi-hop Reasoning Process in NLU from the View of  Bayesian Probability

Despite the recent success of language models (LMs) in natural language understanding (NLU), there are growing concerns about LMs' lack of logical reasoning abilities resulting in poor generalization and robustness. Faced with these concerns, neuro-symbolic integration may be a solution, which guides neural networks to understand the whole reasoning process by symbolic reasoners. In this paper, we propose a generative-symbolic model to explore how effectively this neuro-symbolic model benefits natural language understanding. The experiment on the CLUTRR dataset shows that this neural-symbolic model performs better than the corresponding neural model.

IJCAI 2021 Workshop NSNLI Submissions

Can Cognitive Neuroscience inform Neuro-Symbolic Inference Models?

Perception, Memory, and Inference: The Trinity of Machine Learning

How can BERT Understand High-level Semantics?

Exploring Multi-hop Reasoning Process in NLU from the View of Bayesian Probability

A Generative-Symbolic Model for Logical Reasoning in NLU