Rethinking Data Augmentation for Adversarial Distillation: An Excess Risk Perspective

Di Mi, Yuhang Zhou, Yanjun Zhang, HAIZHUAN YUAN, Zhongyun Hua, Alan Wee-Chung Liew, Leo Yu Zhang

Keywords: Adversarial Robustness Distillation; Adversarial Robustness; Data Augmentation

TL;DR: We study data augmentation in adversarial robustness distillation and propose ASDA, a diffusion-based selection method that surpasses SOTA robustness.

Abstract: Adversarial Robustness Distillation (ARD) enhances the robustness of lightweight models by transferring knowledge from robust teacher models. Most studies focus on output alignment, while input-side augmentation remains underexplored. We reveal a surprising phenomenon: augmentation techniques such as CutMix and AutoAugment, which work well in standard Knowledge Distillation (KD), are ineffective in ARD and can even reduce student robustness. To explain this, we derive an excess risk bound for ARD based on uniform stability, revealing how augmentation diversity and teacher performance on augmented data jointly affect generalization. Our analysis shows that, while augmentation improves sample diversity and smooths the loss landscape, low-quality or overly strong augmentations can compromise teacher reliability during training. This insight highlights a fundamental trade-off in ARD: effective augmentation must balance diversity with teacher reliability. To achieve this balance, we propose ASDA (Active Selection for Diffusion-based Augmentation), which leverages diffusion-generated samples and actively selects informative and teacher-reliable data, guided by output fidelity and entropy. Experiments on CIFAR-10/100 show that ASDA outperforms baselines and surpasses SOTA, clarifying the role of augmentation in ARD and providing a practical solution for improving student robustness.

Primary Area: alignment, fairness, safety, privacy, and societal considerations

Submission Number: 11424