Modeling Knowledge as Functionals for Knowledge Reasoning

Zhengqi Pei, Shuhui Wang, Qingming Huang

Keywords: semantic analysis, knowledge reasoning, commonsense reasoning

TL;DR: We build a functional representation for knowledge and achieve better implementation in knowledge reasoning.

Abstract:

A bottleneck for developing general artificial intelligence is empowering machines with knowledge-reasoning capabilities to facilitate NLP tasks such as semantic search, reading comprehension, and question-answering.
Prior arts focus on integrating distributed knowledge embeddings and representations of pre-trained neural language models to produce outputs; however, there are still large areas for improvement in performance, explainability, and sustainability.
In this paper, we propose to represent ${\bf K}$nowledge ${\bf as}$ the ${\bf F}$unctional representation (${\it KasF}$) with a dynamics-based mechanism that simulates the semantic flow amongst tokens to facilitate knowledge reasoning.
The method utilizes a superposition of semantic fields to represent knowledge by building a dynamical mechanism to compute the similarity between semantic units.
This mechanism comprehensively captures the semantic features and eliminates ambiguities in representing entities and relations.
We first evaluate our ${\it KasF}$ on the WikiQA dataset to demonstrate its superiority in capturing semantic patterns.
Next, we evaluate our ${\it KasF}$ modules on the SQuAD2.0 dataset by replacing the last layer of pre-trained language models fine-tuned on this dataset.
We observe consistent improvements in accuracy with fewer parameters.
Then we evaluate ${\it KasF}$ on the CommonsenseQA benchmark.
On the official blind test set, we achieve state-of-the-art with a single model,
outperforming the prior best ensemble and single models by $0.4\%$ and $3.1\%$, respectively.
It is worth noting that the prior best single model is $47\times$ larger than ours.
Further experiments also demonstrate that ${\it KasF}$ exhibits superiority in dealing with sophisticated sentences.

Primary Area: representation learning for computer vision, audio, language, and other modalities

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.

Submission Guidelines: I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2024/AuthorGuide.

Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors' identity.

No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.

Submission Number: 823