DRAGNN: A Transition-Based Framework for Dynamically Connected Neural Networks

Lingpeng Kong, Chris Alberti, Daniel Andor, Ivan Bogatyy, David Weiss

Nov 04, 2016 (modified: Nov 04, 2016) ICLR 2017 conference submission readers: everyone
  • Abstract: In this work, we present a compact, modular framework for constructing new recurrent neural architectures. Our basic module is a new generic unit, the Transition Based Recurrent Unit (TBRU). In addition to hidden layer activations, TBRUs have discrete state dynamics that allow network connections to be built dynamically as a function of intermediate activations. By connecting multiple TBRUs, we can extend and combine commonly used architectures such as sequence-to-sequence, attention mechanisms, and recursive tree-structured models. A TBRU can also serve as both an {\em encoder} for downstream tasks and as a {\em decoder} for its own task simultaneously, resulting in more accurate multi-task learning. We call our approach Dynamic Recurrent Acyclic Graphical Neural Networks, or DRAGNN. We show that DRAGNN is significantly more accurate and efficient than seq2seq with attention for syntactic dependency parsing and yields more accurate multi-task learning for extractive summarization tasks.
  • TL;DR: Modular framework for dynamically unrolled neural architectures improves structured prediction tasks
  • Keywords: Natural language processing, Deep learning, Multi-modal learning, Structured prediction
  • Conflicts: cmu.edu, google.com