Watermarking for Proprietary Dataset Protection

John Kirchenbauer, Brian R. Bartoldson, Bhavya Kailkhura, Tom Goldstein

Keywords: dataset inference, watermarking, language models

TL;DR: We study watermark based approaches to dataset inference.

Abstract: A growing body of literature suggests that training data membership inference problems are fundamentally hard tasks in modern language modeling settings. We argue that output watermarking techniques are the right gadget to make training membership tests for generative models more tractable, based on prior results showing that language models exhibit residual watermark ``radioactivity'' under partially watermarked training datasets. We pit a watermark-based dataset inference approach head-to-head against traditional loss-based membership inference methods and show that watermarking can achieve comparable membership detection performance when subset exposure is high enough, under an alternate set of assumptions.

Email Sharing: We authorize the sharing of all author emails with Program Chairs.

Data Release: We authorize the release of our submission and author names to the public in the event of acceptance.

Submission Number: 207