Self-distilled BERT with Instance Weights for Denoised Distantly Supervised Relation Extraction

Anonymous

Abstract: The widespread existence of wrongly-labeled instances is a challenge to distantly supervised relation extraction. Most of the previous works use the features from the final output of the encoder and are trained in a bag-level setting. However, intermediate layers of BERT encode a rich hierarchy of linguistic information which is helpful in identifying wrongly-labeled instances. Besides, sentence-level training better utilizes the information than bag-level training, as long as combined with effective noise alleviation. In this work, we design a novel instance weighting mechanism integrated with the self-distilled BERT backbone to enable denoised sentence-level training. Our method aims to alleviate noise and prevent overfitting through dynamic adjustment of learning priorities during self-distillation. Experiments on both held-out and manual datasets indicate that our method achieves state-of-the-art performance and consistent improvements over the baselines.

Paper Type: long

Research Area: Information Extraction