SPEAR++: Scaling Gradient Inversion via Sparse Dictionary Learning

Alexander Bakarsky, Dimitar Iliev Dimitrov, Maximilian Baader, Martin Vechev

Keywords: Federated Learning, Gradient Inversion, Gradient Leakage, Dictionary Recovery, Dictionary Learning, Non-Convex Optimization, Sparsity

TL;DR: We scale exact gradient leakage via complete sparse dictionary learning. In particular, we compare SotA dictionary learning algorithms in various settings to find which performs best for gradient leakage.

Abstract: Federated Learning has seen an increased deployment in real-world scenarios recently, as it enables the distributed training of machine learning models without explicit data sharing between individual clients. Yet, the introduction of the so-called gradient inversion attacks has fundamentally challenged its privacy preserving properties. Unfortunately, as these attacks mostly rely on direct data optimization without any formal guarantees, the vulnerability of real world systems remains in dispute and requires tedious testing for each new federated deployment. To overcome these issues, recently the SPEAR attack was introduced, which is based on a theoretical analysis of the gradients of linear layers with ReLU activations. While SPEAR is an important theoretical breakthrough, the attack’s practicality was severely limited by its exponential runtime in the batch size $b$. In this work, we fill this gap by applying State-of-the-Art techniques from Sparse Dictionary Learning to make the problem of gradient inversion on linear layers with ReLU activations tractable. Our experiments demonstrate that our new attack, SPEAR++, retains all desirable properties of SPEAR, such as robustness to DP noise and FedAvg aggregation, while being applicable to 10x bigger batch sizes.

Submission Number: 78