Replication study of "Privacy-preserving Collaborative Learning with Automatic Transformation Search"Download PDF

Published: 11 Apr 2022, Last Modified: 05 May 2023RC2021Readers: Everyone
Keywords: federated learning, deep learning, privacy, reproducibility, gradient invert, augmentations, defence mechanism
TL;DR: Reproduction of "Privacy-preserving Collaborative Learning with Automatic Transformation Search"
Abstract: 1 Reproducibility Summary 2 Scope of Reproducibility 3 We evaluate the reproducibility of this paper, which proposes an automatic search algorithm to find privacy preserving 4 transformation policies in the setting of federated learning. To achieve this we test all the main claims made by the 5 authors by rerunning the experiments and reporting the reproduced results. We further extend their work to a new 6 dataset. 7 Methodology 8 We perform all experiments using the model architectures and hyperparameters proposed by the authors. We use 9 the same datasets and extend their work to include one new dataset. A codebase was available which enables us to 10 reproduce some of the results. However we deliver a contribution by fully re-implementing the codebase in PyTorch 11 Lightning to ensure all components are modular, and experiments can be easily executed and extended, to the benefit of 12 future research using the authors’ method. All experiments are performed on Nvidia GTX 1080 GPUs. 13 Results 14 Overall we find the same results as the authors: searched transformation policies can defend users in federated learning 15 from reconstruction attacks. These transformations also have negligible impact on training efficiency and model 16 accuracy. However we do not observe the reported correlation between the authors privacy-score and PSNR. We are 17 in contact with the authors about this. Also we find that the results differ greatly from image to image, with standard 18 deviations in PSNR values of over 25% the value. This means that for some specific images the method is not effective. 19 What was easy 20 Paper was clearly written and the general idea was easy to follow. There was a codebase available in PyTorch and part 21 of the experiments were reproducible using this code. 22 What was difficult 23 The codebase was not clearly structured and has to be altered to produce results for most experiments reported in the 24 paper. The reimplementation of the codebase was non-trivial due to otherwise undocumented details in the code having 25 a large impact on outcomes. 26 Communication with original authors 27 The authors were contacted on multiple issues regarding implementation details and notation in the paper. Most of 28 these were resolved swiftly and constructively. On two issues we remain in contact with the authors at this time.
Paper Url:
Paper Venue: CVPR 2021
5 Replies