Local-Forward: Towards Biological Plausibility in Deep Reinforcement Learning

Jonas Guan, Shon Eduard Verch, Claas A Voelcker, Ethan C Jackson, Nicolas Papernot, William A Cunningham

Primary Area: reinforcement learning

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.

Keywords: biological plausibility, deep Q-learning, TD learning

Submission Guidelines: I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2024/AuthorGuide.

TL;DR: We propose a new TD-learning algorithm that is more biologically plausible than DQNs, and show it can achieve comparable performance on the MinAtar testbed.

Abstract:

A lasting critique of deep learning as a model for biological intelligence and learning is the biological implausibility of backpropagation.
Backpropagation requires caching local outputs and propagating a global error via derivatives, neither of which are known to be implemented by biological neurons.
In reinforcement learning, building more biologically plausible agents would allow us to better model human cognition and social behavior, and improve computational efficiency.
We propose Local-Forward, a new temporal-difference learning algorithm (and associated architecture) that trains neural networks to predict Q-values.
Rather than backpropagating error derivates, we rely on updates that are local to each layer of the architecture and additionally use forward connections in time to pass information from upper layers to lower layers via activations. Our approach builds on the recently proposed Forward-Forward algorithm, as well as recurrence and attention in neural architectures.
This approach no longer suffer the aforementioned contradictions with biology.
Furthermore, as a proof-of-concept, we train reinforcement learning agents with Local-Forward to solve control tasks in the MinAtar environments, and show that our method's potential warrants further investigation because it opens avenues for more computational efficient training.

Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors' identity.

No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.

Submission Number: 8802