Low-Rank Adversarial PGD Attack

ICLR 2026 Conference Submission20943 Authors

Keywords: low-rank, adversarial attack, adversarial training

Abstract: Adversarial attacks on deep neural networks have become essential tools for studying model robustness, with Projected Gradient Descent (PGD) being widely adopted due to its effectiveness and computational efficiency. In this work, we provide empirical evidence that PGD perturbations are, on average, low-rank, with their magnitude concentrated in the bottom part of the singular value spectrum across CIFAR-10 and ImageNet datasets and multiple architectures. Building on this insight, we introduce LoRa-PGD, a simple low-rank variation of PGD that directly computes adversarial attacks with controllable rank. Through extensive experiments on different datasets and models from the RobustBench ModelZoo, we demonstrate that LoRa-PGD, systematically outperforms or matches standard PGD in terms of robust accuracy and achieves performance comparable to AutoAttack while requiring orders of magnitude less computational time. Additionally, we show that models adversarially trained with LoRa-PGD, are consistently more accurate and more robust against full-rank attacks compared to standard adversarial training, suggesting that low-rank perturbations capture important but otherwise hidden vulnerability patterns.

Primary Area: applications to computer vision, audio, language, and other modalities

Submission Number: 20943