Direct Feedback Alignment for Recurrent Neural Networks
Keywords: recurrent neural networks, learning algorithms, neuromorphic systems, physically-implementable systems
TL;DR: We develop a version of Direct Feedback Alignment for recurrent neural networks, which allows parallel update of the parameters.
Abstract: Time series and sequential data are widespread in many real-world environments. However, implementing physical and adaptive dynamical systems remains a challenge. Direct Feedback Alignment (DFA) is a learning algorithm for neural networks that overcomes some of the limits of backpropagation and can be implemented in neuromorphic hardware (e.g., photonic accelerators). Until now, DFA has been investigated mainly for feedforward architectures. We adapt DFA for both ``vanilla'' and gated recurrent networks. Unlike backpropagation, the update rule of our DFA can be applied in parallel across time steps, thus removing the sequential propagation of errors. We benchmark DFA on 4 datasets for sequence classification tasks. Although backpropagation still achieves a better predictive accuracy, our DFA shows promising results, especially for environments and physical systems where backpropagation is unavailable.
Submission Number: 6
Loading