Using Reinforcement Learning to Investigate Neural Dynamics During Motor Learning
Keywords: Computational Neuroscience, Motor Learning, RNN modeling, Reinforcement LearningRecent work characterized shifts in preparatory activity of the motor cortex during motor learning. The specific geometry of the shifts during learning, washout, and relearning blocks was hypothesized to implement the acquisition, retention, and retrieval of motor memories. %This leads to the question: what learning algorithms lead to the emergence of these phenomena when monkeys perform a curl field (CF) adaptation task? We sought to develop a framework to train recurrent neural network (RNN) models that could be used to study these motor learning phenomena. We built an environment for a curl field (CF) motor learning task and trained RNNs with reinforcement learning (RL) with novel regularization terms to perform behaviorally realistic reaching trajectories over the course of learning. Our choice of RL rather than supervised learning was motivated by the idea that motor adaptation to a novel environment is a process of reoptimization. We find these models, despite lack of supervision, reproduce many behavioral findings from human and monkey CF adaptation experiments. Relearning is faster than initial learning, indicating formation of motor memories. Optimal reaches under a CF are not straight, but rather curved, which is optimal and has been observed in humans and macaques. These models also captured key neurophysiological findings. We found that the model’s preparatory activity shifted uniformly, independently of the distance to the CF trained target. %We also found the washout shift was consistently approximately orthogonal to the learning shift. Finally, we found that the washout shift becomes more orthogonal to the learning shift when the RNNs are pretrained to learn CF dynamics. We argue the increased fit to neurophysiological recordings is driven by more generalizable circuitry in the pretrained model. This suggests that some aspects of the neural geometry underlying motor memory may be influenced by priors learned over experience in the motor cortex circuitry. Together, this work provides a modeling framework for exploring algorithms that support motor memory, acquisition, retention, and retrieval during motor learning. %These results may inform additional future theoretical exploration of the algorithms underlying motor memory acquisition, retention, and retrieval.
Abstract: Recent work characterized shifts in preparatory activity of the motor cortex during motor learning.
The specific geometry of the shifts during learning, washout, and relearning blocks was hypothesized to implement the acquisition, retention, and retrieval of motor memories.
We sought train recurrent neural network (RNN) models that could be used to study these motor learning phenomena.
We built an environment for a curl field (CF) motor learning task and trained RNNs with reinforcement learning (RL) with novel regularization terms to perform behaviorally realistic reaching trajectories over the course of learning.
Our choice of RL rather than supervised learning was motivated by the idea that motor adaptation to a novel environment, in the absence of demonstrations, is a process of reoptimization.
We find these models, despite lack of supervision, reproduce many behavioral findings from human and monkey CF adaptation experiments.
Relearning is faster than initial learning, indicating formation of motor memories.
Optimal reaches under a CF are not straight, but rather curved, which is optimal and has been observed in humans and macaques.
These models also captured key neurophysiological findings.
We found that the model’s preparatory activity existed in a force-predictive subspace that remained stable across learning, washout, and relearning.
Additionally, preparatory activity shifted uniformly, independently of the distance to the CF trained target.
Finally, we found that the washout shift became more orthogonal to the learning shift, and hence more brain-like, when the RNNs are pretrained to have prior experience with CF dynamics.
We argue the increased fit to neurophysiological recordings is driven by more generalizable and structured dynamical motifs in the model with prior experience from pretraining.
This suggests that the near-orthogonality of learning-washout neural geometry underlying motor memory may be influenced by structured dynamical motifs in the motor cortex circuitry developed from prior experience.
Together, our work takes a step towards elucidating the factors that support motor memory, acquisition, retention, and retrieval during motor learning.
Primary Area: applications to neuroscience & cognitive science
Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
Submission Guidelines: I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.
Reciprocal Reviewing: I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.
Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.
No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.
Submission Number: 13156
Loading