Sample-wise Adaptive Weighting for Transfer Consistency in Adversarial Distillation

ICLR 2026 Conference Submission15072 Authors

Keywords: Adversarial Training, Adversarial Distillation, Robust Saturation

TL;DR: Stronger teachers don't guarantee robust students. We find the key is adversarial transferability. Our method, SAAD, reweights samples based on this insight to improve robustness at no extra cost.

Abstract: Adversarial distillation aims to transfer robustness from a large, robust teacher network to a compact student. However, existing work often neglects to incorporate state-of-the-art robust teachers. Through extensive analysis, we find that stronger teachers do not necessarily yield more robust students–a phenomenon known as robust saturation. While typically attributed to capacity gaps, we show that such explanations are incomplete. Instead, we identify adversarial transferability–the fraction of student-crafted adversarial examples that remain effective against the teacher–as a key factor in successful robustness transfer. Based on this insight, we propose Sample-wise Adaptive Adversarial Distillation (SAAD), which reweights training examples by their measured transferability without incurring additional computational cost. Experiments on CIFAR-10, CIFAR-100, and Tiny-ImageNet show that SAAD consistently improves AutoAttack robustness over prior methods.

Primary Area: alignment, fairness, safety, privacy, and societal considerations

Submission Number: 15072