Robust Multi-bit Text Watermark with LLM-based Paraphrasers

Xiaojun Xu, Jinghan Jia, Yuanshun Yao, Yang Liu, Hang Li

Abstract: We propose an imperceptible multi-bit text watermark embedded by paraphrasing with LLMs. We fine-tune a pair of LLM paraphrasers that are designed to behave differently so that their paraphrasing difference reflected in the text semantics can be identified by a trained decoder. To embed our multi-bit watermark, we use two paraphrasers alternatively to encode the pre-defined binary code at the sentence level. Then we use a text classifier as the decoder to decode each bit of the watermark. Through extensive experiments, we show that our watermarks can achieve over 99.99% detection AUC with small (1.1B) text paraphrasers while keeping the semantic information of the original sentence. More importantly, our pipeline is robust under word substitution and sentence paraphrasing perturbations and generalizes well to out-of-distributional data. We also show the stealthiness of our watermark with LLM-based evaluation.

Lay Summary: We propose to inject watermarks into a piece of text by rewriting them. The rewritten version of the text will have same semantic meaning with the original text, while the watermark code can be decoded from the text using a decoder model. The core technical problem of our method is how to have (1) a good rewriter that can inject the watermark while keeping text meaning and (2) a good decoder that can decode the injected watermark code. In our work, both the rewriter and the decoder are implemented using large language model-based models. The two models are trained together, with the training goal defined to improve both detectabililty (i.e. the watermark can be decoded) and fidelity (i.e. the rewritten text has same semantic meaning). Through experiments, we show that the watermark can be detected in over 99.99% cases and it is hard to tell the difference between rewritten sentence and original sentence from human eyes.

Link To Code: https://github.com/xiaojunxu/multi-bit-text-watermark

Primary Area: Deep Learning->Generative Models and Autoencoders

Keywords: Text Watermark, LLM

Submission Number: 5651