Open Peer Review. Open Publishing. Open Access. Open Discussion. Open Directory. Open Recommendations. Open API. Open Source.
Lookahead Convolution Layer for Unidirectional Recurrent Neural Networks
Chong Wang, Dani Yogatama, Adam Coates, Tony Han, Awni Hannun, Bo Xiao
Feb 18, 2016 (modified: Feb 18, 2016)ICLR 2016 workshop submissionreaders: everyone
Abstract:Recurrent neural networks (RNNs) have been shown to be very effective for many
sequential prediction problems such as speech recognition, machine translation, part-of-speech tagging, and others.
The best variant is typically a bidirectional RNN that learns
representation for a sequence by performing a forward and a backward pass through the entire sequence.
However, unlike unidirectional RNNs, bidirectional RNNs
are challenging to deploy in an online and low-latency setting (e.g., in a speech recognition system),
because they need to see an entire sequence before making a prediction.
We introduce a lookahead convolution layer that incorporates information from future subsequences
in a computationally efficient manner to improve unidirectional recurrent neural networks.
We evaluate our method on speech recognition tasks for two languages---English and Chinese.
Our experiments show that the proposed method outperforms vanilla unidirectional
RNNs and is competitive with bidirectional RNNs in terms of character and word error rates.
Enter your feedback below and we'll get back to you as soon as possible.