TOFU: Towards Obfuscated Federated Updates by Encoding Weight Updates into Gradients from Proxy Data
Abstract: Advances in Federated Learning and an abundance of user data have enabled rich collaborative learning between multiple clients, without sharing user data. This is done via a central server that aggregates learning in the form of weight updates. However, this comes at the cost of repeated expensive communication between the clients and the server, and concerns about compromised user privacy. The inversion of gradients into the data that generated them is termed data leakage. Encryption techniques can be used to counter this leakage, but at added expense. To address these challenges of communication efficiency and privacy, we propose TOFU, a novel algorithm which generates proxy data that encodes the weight updates for each client in a weighted summation of its gradients. Instead of weight updates, this proxy data is now shared. Since input data is far lower in dimensional complexity than weights, this encoding allows us to send much lesser data per communication round. Additionally, the proxy data resembles noise, and even perfect reconstruction from data leakage attacks would invert the decoded gradients into unrecognizable noise, enhancing privacy. We first show that learning is possible via only proxy data that does not resemble real images. In a non-distributed setting, our algorithm achieves an average of 3% accuracy drop on the CIFAR-10 dataset while enabling an average of 17x savings in communication costs. To further improve efficiency and enforce a difference between the synthetic and real data distribution, we show that we can encode the gradient well with downsampled synthetic images achieving an accuracy drop of 3.5% at 65x communication efficiency on average. We then extend this to the federated setup, wherein we recover any accuracy drop via just a few rounds of expensive encrypted gradient exchange. This enables us to learn to near-full accuracy (under 1% drop on average) in a federated setup, while being an average of 4.6x and 6.8x more communication efficient than the standard Federated Averaging algorithm on MNIST and CIFAR-10, respectively.
Submission Length: Regular submission (no more than 12 pages of main content)
Changes Since Last Submission: We have updated the supplementary section A.3.5, A.4 and A.5 to answer reviewer questions about privacy claims and computation time.
Assigned Action Editor: ~Laurent_Massoulié1
Submission Number: 734
Loading