Unveiling Structural Memorization: Structural Membership Inference Attack for Text-to-Image Diffusion Models

Qiao Li, Xiaomeng Fu, Xi Wang, Jin Liu, Xingyu Gao, Jiao Dai, Jizhong Han

Abstract: With the rapid advancements of large-scale text-to-image diffusion models, various practical applications have emerged, bringing significant convenience to society. However, model developers may misuse the unauthorized data to train diffusion models. These data are at risk of being memorized by the models, thus potentially violating citizens' privacy rights. Therefore, in order to judge whether a specific image is utilized as a member of a model's training set, Membership Inference Attack (MIA) is proposed to serve as a tool for privacy protection. Current MIA methods predominantly utilize pixel-wise comparisons as distinguishing clues, considering the pixel-level memorization characteristic of diffusion models. However, it is practically impossible for text-to-image models to memorize all the pixel-level information in massive training sets. Therefore, we move to the more advanced structure-level memorization. Observations on the diffusion process show that the structures of members are better preserved compared to those of nonmembers, indicating that diffusion models possess the capability to remember the structures of member images from training sets. Drawing on these insights, we propose a simple yet effective MIA method tailored for text-to-image diffusion models. Extensive experimental results validate the efficacy of our approach. Compared to current pixel-level baselines, our approach not only achieves state-of-the-art performance but also demonstrates remarkable robustness against various distortions.

Primary Subject Area: [Generation] Social Aspects of Generative AI

Relevance To Conference: The target model of our method is the text-to-image diffusion model, which is one of the typical multimedia generative models and propels the development of Artificial Intelligence Generated Content (AIGC). We first explore the preservation of image structure throughout the diffusion process. Specifically, we observe that image structures remain intact during the initial stages of diffusion, regardless of variations in textual prompts. However, as the diffusion progresses, these structures gradually deteriorate, with textual prompts playing a significant role in this degradation. These findings offer valuable insights for a deeper comprehension of diffusion models. Then based on our findings, we propose an innovative membership inference attack method for text-to-image diffusion models. Our proposed method effectively addresses the unauthorized utilization of images in the training of models, thereby bolstering the protection of image owners' privacy and copyright. In conclusion, our paper contributes to enhancing the privacy and security fairness of the generative AI models.

Submission Number: 2850