Revisiting the Knowledge Injection Frameworks

Anonymous

Abstract: In recent years, pre-trained language models (PLMs) have achieved some eye-catching results on many natural language processing (NLP) tasks.Upon that, a plethora of knowledge-injected PLMs --- assisted by external knowledge graphs --- have been proposed to further enhance or adapt original PLMs on specific downstream tasks.Among these exciting results, we may identify some (potentially) strange odd phenomena such as the imbalance across downstream tasks, little correlation between the injected knowledge and the chosen tasks, the mismatch between knowledge and sentences, etc.These phenomena concern us about the effect of the specific injected knowledge on the model while doing the downstream task.In this work, we intend to comprehensively revisit a series of well-known knowledge-injected frameworks on most common benchmarks, by conducting extensive ablation and control experiments that were previously mostly omitted.Coupled with dense analysis by tracking down the transfer path of the knowledge vectors, we may draw a frustrating conclusion that the current knowledge-injected frameworks may have minimal effect in leveraging the injected knowledge.We further cast a hypothesis to interpret the performance enhancement of the knowledge-injected PLMs from a data augmentation perspective.

Paper Type: long