A Multi-modal Spiking Meta-learner with Brain-Inspired Task-Aware Modulation Scheme

Jun Niu, Zhaokun Zhou, Kaiwei Che, Li Yuan

Abstract: Meta-learning in Spiking Neural Networks (SNNs) has gained increasing attention because it can process data in two modalities, static image and neuromorphic data. However, recent spiking meta-learning works process each few-shot learning task separately, which requires separate training and sometimes different network architectures for each task, and the timesteps of tasks between two modalities are difficult to unify, imposing limitations on their applicability. To address these limitations, we propose Spiking Multi-task Integrated LEarner (SMILE), a unified framework that could process static image tasks and neuromorphic data tasks concurrently, and compute prototypes for each task to better adapt to multi-modal setting. During training, we observe the spiking rate decay phenomenon, the firing rate decreases as network depth increases, leading to training a compromised network. To mitigate spiking rate decay, we propose a brain-inspired task-aware modulation scheme for SMILE, where we modulate the membrane potential of spiking neurons, naming Voltage ModuLation (VML). We conduct experiments on our proposed framework from different perspectives, and experimental results show that our proposed framework achieves state-of-the-art results under various settings.