S3ANformer:Quantized Cognitive Spiking Transformer with S3ANAttention and Feature-refined Feedforward Network

Yunhua Chen

Abstract: Splkng Neural Nework SNNs wth ther bio ispired and low power charctersits ofer a promsing verue fo reating eneroy efidien Transformer architertures However urentSplkingTransormers generaly acea trade-ofbetwen arcurayand camputational fcengy Thisindludes severa halenges quantized splking neurons mprove computatonalficeney butead to aurasdegradation,self atention based on binary spikes sufers from an information boteneck and traditional spking MutiL ayer Perceptrons have nsufrient featurexraction capabites. To adreshese sues we frst ntroduce aearable leak acter and a cordtional actvaton mechanism t ucestuly simulate the forgeting propertes and temporal namics of biolgrcal neurons proposinghe Quanized Cognitive ely ntegrate nd Fire (C). Subseguenly we apy RRel to enhance nfornation represeniaton and design a novel es alued nd spke based yrergistic sfaenin mechanim,called S3AN which improyes acuray wtbout ncreaing computaina complexity Fnaly we propopsea feature refined feforvard network SGRe N. to replace he SMLP Theexperimental results demonstrate that S3ANformer outperforms existing spiking Transformers on various datasets, even achieving 79.84% accuray on ImageNet.