Safely Knowledge Transfer from Source Models via an Iterative Pruning Based Learning Approach

Xiaoyu Sean Lu, Jianan Zhang, Siya Yao, Bo Huang

Abstract: Transfer learning has become a key technique in deep learning, widely adopted in the industry and academia for developing customized models, especially for specific and downstream tasks solving. Despite the prevalence of transfer learning, the target model can easily inherit defects from the source model during the learning process, such as vulnerability to backdoor attacks and adversarial attacks. Thus, this work proposes a novel approach, iterative pruning learning approach (IPLA), that reduces the inheritance of potential defects during the transfer learning process. In order to reduce the vulnerability to attacks and improve the robustness of target model, IPLA evaluates the importance of weights from the source model and retains ones that are critical to the target task, then prunes the redundant weights through an iterative pruning process. Experiments are performed on 4 datasets over 2 backbone source models. Results demonstrate the satisfactory performance of our proposed method.