Patch-Wise Random and Noisy CutMix for Privacy-Preserving Split Learning with Vision Transformer
Abstract: In computer vision, the vision transformer (ViT) has increasingly superseded the convolutional neural network (CNN) for improved accuracy and robustness. Since ViT often comes with large model sizes and high sample complexity, split learning (SL) is a promising solution to training ViT using large memory and computing resources at a server with the sheer amount of private data owned by users or clients. In SL, a ViT is split into two parts under a server-client architecture. The sever stores its upper segment that is associated with multiple clients each of which stores the lower segment. At the cut layer between the upper and lower segments, SL exchanges the cut-layer hidden activations in the forward propagation (FP), referred to as smashed data, and the cut-layer gradients in the backpropagation (BP), which are exposed to various attacks on private training data. To mitigate the risk of data breaches in classification tasks, inspired from the CutMix regularization, we propose a novel privacy-preserving SL framework that injects Gaussian noise into smashed data and mixes randomly chosen patches of smashed data across clients, coined DP-CutMixSL. By analysis, we prove that DP-CutMixSL is a differentially private (DP) mechanism amplifying the privacy budget with respect to membership inference attacks in FP. By simulation, we additionally show that DP-CutMixSL protects privacy from reconstruction attacks in FP and from label inference attacks in BP. Surprisingly, DP-CutMixSL even improves accuracy and robustness to imbalanced data distributions over clients, due to the regularization effect of its patch-wise random CutMix operations.
Submission Length: Long submission (more than 12 pages of main content)
Changes Since Last Submission: Following the suggestions of the reviewers, we have changed the overall structure of the manuscript in this version, especially for Sections 3 and 4 and the appendices. We have also refined the description of privacy attacks, added missing assumptions, and added/changed the figures for the overall structure of DP-CutMixSL and the DP bounds for the updated parameters.
Assigned Action Editor: ~Kangwook_Lee1
Submission Number: 838
Loading