Spiking Neural Networks for Continuous Control: Neuromorphic Reinforcement Learning in Conventional Computing

Jessica Hunter, Md Maruf Hossain Shuvo, Krishna Roy

Keywords: Reinforcement Learning, Spiking Neural Networks, Neuromorphic Computing, Continuous Control, Soft Actor-Critic, Actor-Critic Methods, Deep Reinforcement Learning

TL;DR: We developed a spiking neural network version of Soft Actor-Critic (SANSAC) that matches traditional SAC's performance on complex continuous control tasks while being designed for energy-efficient neuromorphic hardware deployment.

Abstract: Reinforcement learning (RL) algorithms have made strides over the past decade applying them to a wide range of problems and control tasks. However, the deployment of RL on neuromorphic hardware for continuous control tasks remains under-validated. Namely it is unclear whether replacing a conventional actor network with a spiking neural network (SNN) affects the performance of an agent before any hardware-specific benefits manifest. We provide a systematic validation of a minimal, neuromorphically viable spiking actor variant of Soft Actor-Critic (SAC) on conventional hardware, establishing a baseline for future neuromorphic RL research. In this paper, we propose the Spiking Actor Network Soft Actor Critic (SANSAC) to address the use of RL frameworks in continuous environments, designed as a framework that can be implemented on neuromorphic hardware. We compare a traditional Soft Actor Critic (SAC) network to SANSAC in a traditional computer. We demonstrate the near equivalent performance of SANSAC and SAC, while addressing the impact of hidden dimensions. Our results demonstrate the viability of SNN based algorithms in complex continuous environments, as well as competitive performance to traditional neural networks in traditional computers, providing a basis to continue exploring the use of SNNs in continuous RL frameworks.

Submission Number: 46