Phase and Amplitude-aware Prompting for Enhancing Adversarial Robustness

Yibo Xu, Dawei Zhou, Decheng Liu, Nannan Wang

Abstract: Deep neural networks are found to be vulnerable to adversarial perturbations. The prompt-based defense has been increasingly studied due to its high efficiency. However, existing prompt-based defenses mainly exploited mixed prompt patterns, where critical patterns closely related to object semantics lack sufficient focus. The phase and amplitude spectra have been proven to be highly related to specific semantic patterns and crucial for robustness. To this end, in this paper, we propose a Phase and Amplitude-aware Prompting (PAP) defense. Specifically, we construct phase-level and amplitude-level prompts for each class, and adjust weights for prompting according to the model's robust performance under these prompts during training. During testing, we select prompts for each image using its predicted label to obtain the prompted image, which is inputted to the model to get the final prediction. Experimental results demonstrate the effectiveness of our method.

Lay Summary:

Deep learning models are often vulnerable to tiny changes in input images, where these changes are so small that humans can’t see them, but can still fool the model. Some recent defenses try to help make better predictions using extra “prompts”, which perform small adjustments to inputs. But most existing methods focus on mixed semantic information, missing the ones most critical for understanding what is in an image.

Our method focuses on phase and amplitude spectra, which carry important semantic information for recognizing objects. We build unique prompts for each class using these spectra. During training, we automatically adjust the prompts based on how well they help the model stay accurate under attacks. At test time, the model chooses the right prompt based on what it sees for each instance.

By focusing on the "right" parts of an image, this method helps the model make more robust predictions, making AI systems more reliable.

Primary Area: Deep Learning->Robustness

Keywords: Adversarial attack, Adversarial robustness, Prompt-based defense

Submission Number: 4795