Compressed Over-parameterized Federated Learning for Multiple Access Channels

TMLR Paper4937 Authors

Abstract: Federated Learning (FL) is a distributed machine learning (ML) paradigm that addresses user data privacy concerns. Here, a global ML model is learned by aggregating local models that were learned over local data at each edge user (also known as a client). Realizing the benefits of FL is challenging, particularly in communication-constrained environments, such as the Internet of Things (IoT) framework and wireless communication characterized by low bandwidth links over wireless physical channels. A well-known FL protocol over such resource-constrained channels is FL over multiple access channels, also known as FL-MAC, where edge users use the transmission medium simultaneously, hence avoiding the need for orthogonal resources. However, the communication bottleneck at the server in FL can still get choked since modern-day neural networks (NNs) are over-parameterized. Over-parameterized neural networks (ONNs) are trained in the lazy training regime, where the model weights of the NN change very slowly across gradient descent epochs. This motivates the use of incremental model weights. Since such updates are highly sparse, this allows for algorithms that employ compressive sensing (CS), thus allowing compressed model update communication. Accordingly, we propose Compressed Over-parameterized Federated Learning over MAC (or COFL-MAC). We employ a common Gaussian sensing matrix as the dictionary to compress the per-user model updates. By means of NTK theory, we show that the COFL-MAC framework exhibits exponential convergence in addition to being communication efficient. Using the CIFAR-10 and FMNIST datasets, we empirically demonstrate that the proposed framework outperforms the gradient compression benchmark strategies - Top-k (with correction), SignSGD, and MQAT, in terms of communication efficiency for a given test accuracy for different data heterogeneity levels among the clients.

Submission Length: Long submission (more than 12 pages of main content)

Previous TMLR Submission Url: https://openreview.net/forum?id=Tm4yYcC3Nb&noteId=Tm4yYcC3Nb

Changes Since Last Submission: I accidentally uploaded an older version of the manuscript previously. Rectified now.

Assigned Action Editor: ~Eduard_Gorbunov1

Submission Number: 4937