Towards Robust Training via Gradient-diversified Backpropagation
Keywords: Adversarial Training, Adversarial Augmentation, Domain Generalization
TL;DR: We purpose the Stochastic Loss Integration Method, which can be integrated to boost existing adversarial training and adversarial augmentation methods.
Abstract: Neural networks are prone to be vulnerable to adversarial attacks and domain shifts. Adversarial-driven methods including adversarial training and adversarial augmentation, have been frequently proposed to improve the model's robustness against adversarial attacks and distribution-shifted samples. Nonetheless, recent research on adversarial attacks has cast a spotlight on the robustness lacuna against attacks targeted at intermediate layers. Towards analyzing the rationale for this robustness lacuna, this paper investigates the layer-wise adversarial effect and adversarial gradients w.r.t intermediate layers. We observe that previous adversarial-driven methods tend to generate limited perturbations in the shallow intermediate layers compared with the deep output layer and there is a domain gap existing between the intermediate layer gradients generated by various adversarial techniques. The observed robustness lacuna can be primarily attributed to the exclusive utilization of loss functions on the output layer for adversarial gradient generation. This inherent practice constrains the adversarial impact on the shallow intermediate layers. Therefore, from the standing point of diversifying the adversarial gradients to ensure sufficient training and robustness of intermediate layers, this paper proposes a novel Stochastic Loss Integration Method (SLIM), which can be instantiated into the existing adversarial-driven methods in a plug-and-play manner. Experimental results across diverse tasks, including classification and segmentation, as well as various areas such as adversarial robustness and domain generalization, validate the effectiveness of our proposed method. Furthermore, we provide an in-depth analysis to offer a comprehensive understanding of layer-wise training involving various loss terms.
Primary Area: transfer learning, meta learning, and lifelong learning
Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
Submission Guidelines: I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2024/AuthorGuide.
Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors' identity.
No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.
Submission Number: 273
Loading