How do Active Dendrite Networks Mitigate Catastrophic Forgetting?

Sankarshan Damle, Satya Lokam, Navin Goyal

Keywords: Continual Learning, Active Dendrite Networks, Interpretability

TL;DR: This paper shows that ADNs reduce catastrophic forgetting in Sparse Parity tasks due to task separation via uncorrelated context vectors, but fail for Modular Addition due to high task correlation.

Abstract: We investigate the efficacy of Active Dendrite Networks (ADNs) in mitigating catastrophic forgetting in Continual Learning (CL). We consider Sparse Parity and Modular Addition to be our CL-task sequences. ADNs mitigate forgetting for Sparse Parity, but not for Modular Addition. For Sparse Parity, we perform an interpretability analysis to highlight the effectiveness of orthogonal (or uncorrelated) context vectors and task vectors in mitigation. We demonstrate that uncorrelated context vectors facilitate the creation of distinct subnetworks within ADNs, aiding in task separation. We also look at task uncorrelatedness to explain the difference in ADN's performance for Sparse Parity and Modular Addition.

Submission Number: 47