Finding a Jack-of-All-Trades: An Examination of Semi-supervised Learning in Reading Comprehension

Rudolf Kadlec, Ondřej Bajgar, Peter Hrincar, Jan Kleindienst

Nov 04, 2016 (modified: Jan 20, 2017) ICLR 2017 conference submission readers: everyone
  • Abstract: Deep learning has proven useful on many NLP tasks including reading comprehension. However it requires a lot of training data which are not available in some domains of application. Hence we examine the possibility of using data-rich domains to pre-train models and then apply them in domains where training data are harder to get. Specifically, we train a neural-network-based model on two context-question-answer datasets, the BookTest and CNN/Daily Mail, and we monitor transfer to subsets of bAbI, a set of artificial tasks designed to test specific reasoning abilities, and of SQuAD, a question-answering dataset which is much closer to real-world applications. Our experiments show very limited transfer if the model isn’t shown any training examples from the target domain however the results are promising if the model is shown at least a few target-domain examples. Furthermore we show that the effect of pre-training is not limited to word embeddings.
  • TL;DR: We examine effect of transfer learning in AS Reader model from two source domains (CNN/DM and BookTest) to two target domains (bAbI and SQuAD).
  • Conflicts:
  • Keywords: Natural language processing, Semi-Supervised Learning, Deep learning, Transfer Learning