Guardians of Image Quality: Benchmarking Defenses Against Adversarial Attacks on Image Quality Metrics

Aleksandr Gushchin, Khaled Abud, Georgii Bychkov, Ekaterina Shumitskaya, Anna Chistyakova, Sergey Lavrushkin, Bader Rasheed, Kirill Malyshev, Dmitriy S. Vatolin, Anastasia Antsiferova

TL;DR: This paper presents a new benchmark of defense methods against adversarial attacks on image quality metrics

Abstract: Modern neural-network-based Image Quality Assessment (IQA) metrics are vulnerable to adversarial attacks, which can be exploited to manipulate search engine rankings, benchmark results, and content quality assessments, raising concerns about the reliability of IQA metrics in critical applications. This paper presents the first comprehensive study of IQA defense mechanisms in response to adversarial attacks on these metrics to pave the way for safer use of IQA metrics. We systematically evaluated 30 defense strategies, including purification, training-based, and certified methods --- and applied 14 adversarial attacks in adaptive and non-adaptive settings to compare these defenses on 9 no-reference IQA metrics. Our proposed benchmark aims to guide the development of IQA defense methods and is open to submissions; the latest results and code are at https://msu-video-group.github.io/adversarial-defenses-for-iqa/.

Lay Summary:

There are a vast collection of computer programs that judge how good an image looks. But tiny, invisible tweaks to an image can fool these programs into giving a bad photo a perfect score (or vice versa). That’s dangerous if websites or benchmarks start trusting these ratings to sort search results, decide which photos to feature, or even flag inappropriate content.

To make these image‑quality judges safer, we tested 30 different “defense” ideas ranging from image filters to new training tricks and even mathematical guarantees and then tried to break them with 14 different attack methods (some that know exactly how the defense works, some that don’t). We ran all of this on nine commonly used, no‑reference IQA tools (those that don’t compare to an original “perfect” image) to see which defenses really hold up under attacks.

Finally, we’ve wrapped all our tests into a public benchmark so anyone can plug in their own defenses and see how they stack up. By standardizing these comparisons, we aim to accelerate the development of IQA systems that can’t be so easily deceived.

Link To Code: https://github.com/msu-video-group/adversarial-defenses-for-iqa

Primary Area: General Machine Learning->Evaluation

Keywords: adversarial defenses, image quality assessment, adversarial attacks, image quality metrics, benchmark

Flagged For Ethics Review: true

Submission Number: 4478