Spatiotemporal motor learning with reward-modulated Hebbian plasticity in modular reservoir computing

Yuji Kawai, Minoru Asada

Abstract: Generation of complex patterns at a specific timing is crucial to most forms of learning and behavior, which are acquired through dopamine-modulated plasticity in the striatum. However, the neural mechanisms of such reward-based spatiotemporal processing remain unknown. Inspired by the cortico-striatal circuits, this study developed a new reservoir computing method, a class of recurrent neural networks, based on reward-modulated Hebbian learning (RMHL) for the spatiotemporal motor learning. We utilized a reservoir of basal dynamics (reBASICS), which generated self-sustained limit cycle oscillations with various frequencies, as a reservoir structure. Then, the oscillations were linearly integrated as readout output, in which readout weights were modulated with RMHL. The simulations showed that reBASICS-based RMHL was able to accomplish both motor timing and pattern drawing tasks, for which existing reservoir-based RMHL failed. Further, introducing an eligibility trace mechanism into RMHL allowed the model to learn motor timing even when reward-based modulation was delayed. In conclusion, this model is proposed as a new computational model of temporal processing of the striatum, where the cortical areas generate stable oscillations. From the oscillatory dynamics, spatiotemporal patterns are learned using RMHL in the striatum.