Hyper adversarial tuning for boosting adversarial robustness of pretrained large vision transformers

Kangtao Lv, Wenyan Fan, Huangsen Cao, Kainan Tu, Yihuai Xu, Zhimeng Zhang, Yang Li, Xin Ding, Yongwei Wang

Abstract: Highlights•Unifies multiple defenses for pretrained large vision Transformers in one adversarial tuning framework.•A HyperNetwork generates LoRA to share defensive knowledge among different methods.•Being extensible and computationally efficient to establish stronger adversarial defenses.•Surpasses full-parameter robust fine-tuning with significantly less computation, achieving a new state-of-the-art.