MMDEND: Dendrite-Inspired Multi-Branch Multi-Compartment Parallel Spiking Neuron for Sequence Modeling

ACL ARR 2025 February Submission1466 Authors

Abstract:

Vanilla spiking neurons are simplified from complex biological neurons with dendrites, soma, and synapses, into single somatic compartments. Due to limitations in performance and training efficiency, vanilla spiking neurons face significant challenges in modeling long sequences.
In terms of performance, the oversimplified dynamics of spiking neurons omit long-term temporal dependencies. Additionally, the long-tail membrane potential distribution and binary activation discretization errors further limit their capacity to model long sequences.
In terms of efficiency, the serial mechanism of spiking neurons leads to excessively long training times for long sequences. Though parallel spiking neurons are an efficient solution, their number of parameters is often tied to the hidden dimension or sequence length, which makes current parallel neurons unsuitable for large architectures.
To address these issues, we propose **MMDEND**: a **M**ulti-Branch **M**ulti-Compartment Parallel Spiking **Dend**ritic Neuron. Its proportion-adjustable multi-branch, multi-compartment structure enables long-term temporal dependencies.
Additionally, we introduce a **S**caling-**S**hifting Integer **F**iring (**SSF**) mechanism that fits the long-tail membrane potential distribution, retains efficiency, and mitigates discretization errors.
Compared with parallel neurons, MMDEND achieves better long-sequence modeling capability with fewer parameters and lower energy consumption. Visualization also confirms that the SSF mechanism effectively fits long-tail distributions.

Paper Type: Long

Research Area: Machine Learning for NLP

Research Area Keywords: Spiking Neurons, Dendritic Neuron Modeling.

Contribution Types: Model analysis & interpretability

Languages Studied: English

Submission Number: 1466