Adaptive Code Watermarking Through Reinforcement Learning

ICLR 2026 Conference Submission15039 Authors

Keywords: Large Language Model, LLM Watermarking, Code Watermarking

TL;DR: We use reinforcement learning to train a watermark model that can effectively embed watermarks in LLM-generated code while preserving code functionality.

Abstract: Protecting intellectual property on LLM-generated code necessitates effective watermarking systems that can operate within code's highly structured, syntactically constrained nature. In this work, we introduce CodeTracer, an innovative adaptive code watermarking framework underpinned by a novel reinforcement learning training paradigm. At its core, CodeTracer features a policy-driven approach that utilizes a parameterized model to intelligently bias token choices during next-token prediction. This strategy ensures that embedded watermarks maintain code functionality while exhibiting subtle yet statistically detectable deviations from typical token distributions. To facilitate policy learning, we devise a comprehensive reward system that seamlessly integrates execution feedback with watermark embedding signals, balancing process-level and outcome-level rewards. Additionally, we employ Gumbel Top-k reparameterization to enable gradient-based optimization of discrete watermarking decisions. Extensive comparative evaluations demonstrate CodeTracer's significant superiority over state-of-the-art baselines in both watermark detectability and the preservation of generated code's functionality. Our code is available at https://anonymous.4open.science/r/CodeTracer-B8EE.

Primary Area: foundation or frontier models, including LLMs

Submission Number: 15039