Adversarial Example Quality Assessment: A Large-scale Dataset and Strong Baseline

Jia-Li Yin, Menghao chen, jin Han, Bo-Hao Chen, Ximeng Liu

Abstract:

Adversarial examples (AEs), which are maliciously hand-crafted by adding perturbations to benign images, reveal the vulnerability of deep neural networks (DNNs) and have been used as a benchmark for evaluating model robustness. With great efforts have been devoted to generating AEs with stronger attack ability, the visual quality of AEs is generally neglected in previous studies. The lack of a good quality measure of AEs makes it very hard to compare the relative merits of attack techniques and is hindering technological advancement. How to evaluate the visual quality of AEs remains an understudied and unsolved problem.
In this work, we make the first attempt to fill the gap by presenting an image quality assessment method specifically designed for AEs. Towards this goal, we first construct a new database, called AdvDB, developed on diverse adversarial examples with elaborated annotations. We also propose a detection-based structural similarity index (AdvDSS) for adversarial example perceptual quality assessment. Specifically, the visual saliency for capturing the near-threshold adversarial distortions is first detected via human visual system (HVS) techniques and then the structural similarity is extracted to predict the quality score. Moreover, we further propose AEQA for overall adversarial example quality assessment by integrating the perceptual quality and attack intensity of AEs. Extensive experiments validate that the proposed AdvDSS achieves state-of-the-art performance which is more consistent with human opinions.

Primary Subject Area: [Content] Media Interpretation

Secondary Subject Area: [Experience] Multimedia Applications

Relevance To Conference: Adversarial attacks are now a trending topic in the multimedia and machine learning community, numerous attack techniques are proposed and used as a benchmark for evaluating model robustness. However, with great efforts have been devoted to generating adversarial examples (AEs) with stronger attack ability, the visual quality of AEs is generally neglected in previous studies. In this work, we make the first attempt to fill the gap by presenting an image quality assessment method specifically designed for AEs. Based on a new constructed database AdvDB, our method propose a detection-based structural similarity index (AdvDSS) for adversarial example perceptual quality assessment. Moreover, we further propose AEQA for overall adversarial example quality assessment by integrating the perceptual quality and attack intensity of AEs. Within the framework of the "Multimedia Interpretation" theme, this work offers a fresh perspective on analyzing and designing adversarial examples, which may lead to new ways of interpreting or creating security-connected multimedia content.

Submission Number: 2563