Anchored Decoding: Provably Reducing Copyright Risk for Any Language Model

Jacqueline He, Jonathan Hayase, Wen-tau Yih, Sewoong Oh, Luke Zettlemoyer, Pang Wei Koh

Keywords: copyright mitigation, copyright risk, two-model decoding, model fusion, verbatim memorization, byte-level decoding

TL;DR: We propose a decoding method that provably reduces the likelihood of generating copyrighted text by interpolating between a model trained on openly licensed data, and a higher-utility model trained on mixed-license data.

Abstract: Modern language models (LMs) tend to memorize portions of their training data and reproduce verbatim spans. When the underlying sources are sensitive or copyright-protected, such reproduction raises issues of consent and compensation for creators and compliance risks for developers. We propose Anchored Decoding, a plug-and-play inference-time method for suppressing verbatim reproduction: it enables decoding from any risky LM trained on mixed-license data by keeping generation in bounded proximity to a permissively trained safe LM. Anchored Decoding does so by adaptively allocating a user-chosen information budget over the generation trajectory and enforcing per-step constraints that yield a sequence-level guarantee, enabling a tunable risk–utility trade-off. To make Anchored Decoding practically useful, we introduce a new permissively trained safe model (TinyComma 1.8B), as well as $\textrm{Anchored}\_{\textrm{Byte}}$ Decoding, a byte-level variant of our method that enables cross-vocabulary fusion via the ByteSampler (Hayase et al., 2025) framework. We evaluate our methods across six model pairs on long-form evaluations of copyright risk and utility. Anchored and $\textrm{Anchored}\_{\textrm{Byte}}$ Decoding define a new Pareto frontier, preserving near-original fluency and factuality while eliminating up to 75% of the measurable copying gap (averaged over six copying metrics) between the risky baseline and a safe reference, at a modest inference overhead.

Email Sharing: We authorize the sharing of all author emails with Program Chairs.

Data Release: We authorize the release of our submission and author names to the public in the event of acceptance.

Submission Number: 83