Fast weight programming and linear transformers: from machine learning to neurobiology

TMLR Paper5599 Authors

Abstract: Recent advances in artificial neural networks for machine learning, and language modeling in particular, have established a family of recurrent neural network (RNN) architectures that, unlike conventional RNNs with vector-form hidden states, use two-dimensional (2D) matrix-form hidden states. Such 2D-state RNNs, known as Fast Weight Programmers (FWPs), can be interpreted as a neural network whose synaptic weights (called fast weights) dynamically change over time as a function of input observations, and serve as short-term memory storage; corresponding synaptic weight modifications are controlled or programmed by another network (the programmer) whose parameters are trained (e.g., by gradient descent). In this Primer, we review the technical foundations of FWPs, their computational characteristics, and their connections to transformers and state space models. We also discuss connections between FWPs and models of synaptic plasticity in the brain, suggesting a convergence of natural and artificial intelligence.

Submission Length: Long submission (more than 12 pages of main content)

Changes Since Last Submission:

Please find the edited text highlighted in blue in the PDF.

The summary of changes is as follows:

- Revised the sentence in paragraph "Practical considerations" (Page 12) regarding the unpublished reinforcement learning results (**Reviewer JBY1 & Reviewer HYzD**)

- Revised the first sentence of the 3rd paragraph in Sec. 2.2 (**Reviewer JBY1**)

- Added a **Glossary** listing key neurobiology terms on Page 19 (**Reviewer JBY1**)

- Added **Appendix A** to provide derivations connecting the update rules and local objective functions, and updated Table 1 to correct errors, missing factors, formatting issues, and typos (**Reviewer JBY1**)

- Corrected a typo: diagonal (**Reviewer JBY1**)

- Added a sentence to state connection between vanilla FWP and SSM in Sec 3.1. 3rd paragraph (**Reviewer HYzD**)

- Added **Footnote 3** to acknowledge that "fast" vs "slow" distinction may not hold in the case of RTRL (**Reviewer HYzD**)

- Added a sentence in Sec 4.2 clarifying that the FWP framework can cover the case in which the slow net and/or the fast net is recurrent (**Reviewer HYzD**)

Assigned Action Editor: ~Robert_Legenstein1

Submission Number: 5599