HisBERT for Conversational Reading Comprehension

Chuang Liu, Deyi Xiong, Yuxiang Jia, Hongying Zan, Changjian Hu

Abstract: It is still an open problem to efficiently explore BERT for contextual question answering, such as conversational reading comprehension (CRC). Previous work on using BERT for CRC does not deeply integrate conversation history into the architecture of BERT. In order to make better use of BERT for CRC, in this paper, we propose HisBERT (BERT with conversation history) that consists of two parallel units and divides BERT into three blocks to better integrate conversation history. Additionally we use adversarial training to disturb the word embedding layer, so as to increase the robustness of the proposed model. Experiment results show that HisBERT not only achieves strong competitive performance, but also benefits from a good trade-off between performance and training time.