CoDeC: Communication-Efficient Decentralized Continual Learning

Sakshi Choudhary, Sai Aparna Aketi, Gobinda Saha, Kaushik Roy

Abstract:

Training at the edge utilizes continuously evolving data generated at different locations. Privacy concerns prohibit the co-location of this spatially as well as temporally distributed data, deeming it crucial to design training algorithms that enable efficient continual learning
over decentralized private data. Decentralized learning allows serverless training with spatially distributed data. A fundamental barrier in such setups is the high bandwidth cost of communicating model updates between agents. Moreover, existing works under this training paradigm are not inherently suitable for learning a temporal sequence of tasks while retaining the previously acquired knowledge. In this work, we propose CoDeC, a novel communication-efficient decentralized continual learning algorithm that addresses these challenges. We mitigate catastrophic forgetting while learning a distributed task sequence by incorporating orthogonal gradient projection within a gossip-based decentralized learning algorithm. Further, CoDeC includes a novel lossless communication compression scheme based on the gradient subspaces. We theoretically analyze the convergence rate for our algorithm and demonstrate through an extensive set of experiments that CoDeC successfully learns distributed continual tasks with minimal forgetting. The proposed compression scheme results in up to 4.8× reduction in communication costs without any loss in performance.

Submission Length: Regular submission (no more than 12 pages of main content)

Video: https://drive.google.com/file/d/1UPp5btj8j_JziRPeyWbEB6f7y_KBFsZY/view?usp=sharing

Code: https://github.com/Sakshi09Ch/CoDeC

Assigned Action Editor: ~Cedric_Archambeau1

Submission Number: 1741