DIS-CO: Discovering Copyrighted Content in VLMs Training Data

André V. Duarte, Xuandong Zhao, Arlindo L. Oliveira, Lei Li

TL;DR: DIS-CO identifies copyrighted content in VLMs training data by showing that models can link movie frames to their titles in a free-form text generation setting, even when the frames are highly challenging, suggesting prior exposure during training.

Abstract: *How can we verify whether copyrighted content was used to train a large vision-language model (VLM) without direct access to its training data?* Motivated by the hypothesis that a VLM is able to recognize images from its training corpus, we propose DIS-CO, a novel approach to infer the inclusion of copyrighted content during the model's development. By repeatedly querying a VLM with specific frames from targeted copyrighted material, DIS-CO extracts the content's identity through free-form text completions. To assess its effectiveness, we introduce MovieTection, a benchmark comprising 14,000 frames paired with detailed captions, drawn from films released both before and after a model’s training cutoff. Our results show that DIS-CO significantly improves detection performance, nearly doubling the average AUC of the best prior method on models with logits available. Our findings also highlight a broader concern: all tested models appear to have been exposed to some extent to copyrighted content. We provide the code in the supplementary materials.

Lay Summary:

Large Vision-Language Models learn by training on *huge collections of images and text*. However, there is *growing concern* that these collections might include **copyrighted material**, raising important legal and ethical questions.

Motivated by the idea that models can *recognize images they have seen before*, we developed **DIS-CO**: a *new approach* for detecting whether a model has **memorized specific visual content**, with a special focus on *copyrighted material*.

**DIS-CO works by repeatedly prompting the model to identify the source of carefully selected images**: for example, asking, *“Which movie is this frame from?”*, and requiring the model to generate its answer *freely*, instead of picking from multiple-choice options. This *free-form format is crucial*: it makes it *extremely unlikely* for the model to guess the correct answer by chance. If the model can *reliably provide the correct titles for challenging frames*, it offers **strong evidence** that it encountered this content during training.

To *rigorously evaluate DIS-CO*, we created **MovieTection**, a *benchmark featuring 14,000 frames from 100 movies*, and our results are clear: *all tested models showed signs of exposure to copyrighted content*. With **DIS-CO**, even when companies do *not disclose their training data*, it becomes possible to reveal whether specific copyrighted material was used. As such, we believe this offers a *new path toward greater transparency and accountability in AI development*.

Link To Code: https://github.com/avduarte333/DIS-CO

Primary Area: Social Aspects->Safety

Keywords: Copyrighted Content Detection, Membership Inference Attacks, Large Vision Language Models, Natural Language Processing

Flagged For Ethics Review: true

Submission Number: 13148