Replication study of "Privacy-preserving Collaborative Learning with Automatic Transformation Search"

Karolina Drabent, Konrad Bereda, Stefan Wijnja, Thijs Sluijter

Keywords: federated learning, deep learning, privacy, reproducibility, gradient invert, augmentations, defence mechanism

TL;DR: Reproduction of "Privacy-preserving Collaborative Learning with Automatic Transformation Search"

Abstract:

1 Reproducibility Summary

2 Scope of Reproducibility
3 We evaluate the reproducibility of this paper, which proposes an automatic search algorithm to find privacy preserving
4 transformation policies in the setting of federated learning. To achieve this we test all the main claims made by the
5 authors by rerunning the experiments and reporting the reproduced results. We further extend their work to a new
6 dataset.

7 Methodology
8 We perform all experiments using the model architectures and hyperparameters proposed by the authors. We use
9 the same datasets and extend their work to include one new dataset. A codebase was available which enables us to
10 reproduce some of the results. However we deliver a contribution by fully re-implementing the codebase in PyTorch
11 Lightning to ensure all components are modular, and experiments can be easily executed and extended, to the benefit of
12 future research using the authors’ method. All experiments are performed on Nvidia GTX 1080 GPUs.

13 Results
14 Overall we find the same results as the authors: searched transformation policies can defend users in federated learning
15 from reconstruction attacks. These transformations also have negligible impact on training efficiency and model
16 accuracy. However we do not observe the reported correlation between the authors privacy-score and PSNR. We are
17 in contact with the authors about this. Also we find that the results differ greatly from image to image, with standard
18 deviations in PSNR values of over 25% the value. This means that for some specific images the method is not effective.

19 What was easy
20 Paper was clearly written and the general idea was easy to follow. There was a codebase available in PyTorch and part
21 of the experiments were reproducible using this code.

22 What was difficult
23 The codebase was not clearly structured and has to be altered to produce results for most experiments reported in the
24 paper. The reimplementation of the codebase was non-trivial due to otherwise undocumented details in the code having
25 a large impact on outcomes.

26 Communication with original authors
27 The authors were contacted on multiple issues regarding implementation details and notation in the paper. Most of
28 these were resolved swiftly and constructively. On two issues we remain in contact with the authors at this time.

Paper Url: https://openaccess.thecvf.com/content/CVPR2021/papers/Gao_Privacy-Preserving_Collaborative_Learning_With_Automatic_Transformation_Search_CVPR_2021_paper.pdf

Paper Venue: CVPR 2021