PoisonedEye: Knowledge Poisoning Attack on Retrieval-Augmented Generation based Large Vision-Language Models

Chenyang Zhang, Xiaoyu Zhang, Jian Lou, Kai Wu, Zilong Wang, Xiaofeng Chen

TL;DR: This paper proposes the first knowledge poisoning attack on retrieval-augmented Large Vision-Language Models.

Abstract: Vision-Language Retrieval-Augmented Generation (VLRAG) systems have been widely applied to Large Vision-Language Models (LVLMs) to enhance their generation ability. However, the reliance on external multimodal knowledge databases renders VLRAG systems vulnerable to malicious poisoning attacks. In this paper, we introduce PoisonedEye, the first knowledge poisoning attack designed for VLRAG systems. Our attack successfully manipulates the response of the VLRAG system for the target query by injecting only one poison sample into the knowledge database. To construct the poison sample, we follow two key properties for the retrieval and generation process, and identify the solution by satisfying these properties. Besides, we also introduce a class query targeted poisoning attack, a more generalized strategy that extends the poisoning effect to an entire class of target queries. Extensive experiments on multiple query datasets, retrievers, and LVLMs demonstrate that our attack is highly effective in compromising VLRAG systems.

Lay Summary:

AI systems have gained the ability to "see" and "describe", such as explaining what is in a picture or answering questions about an image. These systems often use external knowledge bases to improve their performance. However, this reliance on outside information can be a security risk. If the knowledge base is tampered with by attackers, the system's answers may become incorrect or even mislead users.

In this paper, we introduce PoisonedEye, the first attack designed specifically to exploit these vision-language systems. Our method allows an attacker to manipulate the system's response to a specific question by simply inserting one malicious data point into the knowledge base. As a result, when a user asks the specific question, the system will give a wrong or harmful answer that was planned by the attacker.

This research reminds us that many current AI systems that rely on external knowledge might be vulnerable to attacks. We need to pay more attention to securing these systems and improving their defenses.

Link To Code: https://github.com/123000001212/PoisonedEye

Primary Area: Social Aspects->Security

Keywords: poisoning attack, retrieval-augmented generation, vision-language models

Flagged For Ethics Review: true

Submission Number: 3598