Dopamine transients in the ventral striatum provide evidence for average-reward reinforcement learning
Keywords: Average-reward reinforcement learning, Dopamine, Reward prediction error, Ventral striatum
Abstract: Agents in real environments need to organize their behavior over a wide range of time scales. This might be achieved by reinforcement learning (RL) algorithms employing a spectrum of discount factors. Neural evidence for this idea includes recordings of dopamine (DA) release transients, which appear to reflect shorter time horizons in dorsal striatum and much longer horizons in ventral striatum (VS). However, this also presents a challenge, because with very long time horizons all states have similar, large values, impeding learning. Prior theoretical work has therefore proposed algorithms, including average-reward RL, that segregate out the large shared component of value. Here we compare temporal-difference reward prediction errors derived from recurrent neural network models (RNNs) to rat VS DA transients measured in three behavioral tasks. We show that using average-reward RL to train RNNs can provide an improved match to VS DA, compared to using discounting alone. We further find that the activity dynamics in RNNs trained with average-reward RL readily encodes key decision variables such as recent reward history, in a task-specific manner. The functional alignment between DA dynamics and average-reward RL may offer new insights into neural mechanisms of learning and decision-making.
Primary Area: applications to neuroscience & cognitive science
Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
Submission Guidelines: I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.
Reciprocal Reviewing: I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.
Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.
No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.
Submission Number: 8288
Loading