Open Peer Review. Open Publishing. Open Access. Open Discussion. Open Directory. Open Recommendations. Open API. Open Source.
Fast Weight Long Short-Term Memory
T. Anderson Keller, Sharath Nittur Sridhar, Xin Wang
Feb 12, 2018 (modified: Jun 04, 2018)ICLR 2018 Workshop Submissionreaders: everyoneShow Bibtex
Abstract:Associative memory using fast weights is a short-term memory mechanism that substantially improves the memory capacity and time scale of recurrent neural networks (RNNs). As recent studies introduced fast weights only to regular RNNs, it is unknown whether fast weight memory is beneficial to gated RNNs. In this work, we report a significant synergy between long short-term memory (LSTM) networks and fast weight associative memories. We show that this combination, in learning associative retrieval tasks, results in much faster training and lower test error, a performance boost most prominent at high memory task difficulties.
Keywords:Fast Weights, LSTM, Long Short-Term Memory, Associative Memory, Recurrent, RNN
TL;DR:We show that LSTM with fast weight associative memory trains much faster and achieves lower test error in associative retrieval tasks than previously reported fast weights RNNs.
Enter your feedback below and we'll get back to you as soon as possible.