Few-shot Logical Form Generation for KGQA via Pseudo Label Pre-training and Knowledge Distillation

Anonymous

TL;DR: In this work, we propose a novel pseudo logical form pre-training strategy to separate the learning process into a pre-training stage and a fine-tuning stage and

Abstract:

With the emergence of neural language models, extensive research has been conducted on question-answering systems. Knowledge Graph Question Answering (KGQA) remains a hot topic because it returns answers from reliable knowledge graphs, while language models sometimes suffer from hallucinations and produce unfaithful answers. An intuitive and explicable solution for KGQA involves generating logical forms (such as SPARQL, s-expression, Cypher, etc.) that can be executed against the KG. However, due to the heterogeneous nature of KG schemas across different KGs, distinct logical forms are required, thereby necessitating various models. The training process for such models to adapt to diverse KG schema settings is resource-intensive or data-hungry when built upon large language models or smaller models, respectively.
In this work, we propose a novel pseudo logical form pre-training strategy to separate the learning process into a pre-training stage and a fine-tuning stage. In the pre-training stage, the model learns to generate KG items according to its understanding of the question. In the fine-tuning stage, the model may focus on learning logical form grammar with limited labeled data. Besides, the proposed strategy can be combined with knowledge distillation to further boost the model's performance. Experimental evaluations conducted on MetaQA and KQA Pro show that our model outperforms several strong baselines, thus substantiating the efficacy of our proposed techniques.

Paper Type: long

Research Area: Question Answering

Contribution Types: Approaches to low-resource settings

Languages Studied: english