MemBench: Memorized Image Trigger Prompt Dataset for Diffusion Models

Chunsan Hong, Tae-Hyun Oh, Minhyuk Sung

Abstract: Diffusion models have achieved remarkable success in Text-to-Image generation tasks, leading to the development of many commercial models. However, recent studies have reported that diffusion models often repeatedly generate memorized images in train data when triggered by specific prompts, potentially raising social issues ranging from copyright to privacy concerns. To sidestep the memorization, recent studies have been conducted to develop memorization mitigation methods for diffusion models. Nevertheless, the lack of benchmarks hinders the assessment of the true effectiveness of these methods. In this work, we present MemBench, the first benchmark for evaluating image memorization mitigation methods. Our benchmark includes a large number of memorized image trigger prompts in various Text-to-Image diffusion models. Furthermore, in contrast to the prior work evaluating mitigation performance only on trigger prompts, we present metrics evaluating on both trigger prompts and general prompts, so that we can see whether mitigation methods address the memorization issue while maintaining performance for general prompts. Through our MemBench evaluation, we revealed that existing memorization mitigation methods notably degrade the overall performance of diffusion models and need to be further developed.

Submission Length: Regular submission (no more than 12 pages of main content)

Code: https://github.com/chunsanHong/MemBench_code

Assigned Action Editor: ~Li_Dong1

Submission Number: 4177