DP-RFT: Learning to Generate Synthetic Text via Differentially Private Reinforcement Fine-tuning

ICLR 2026 Conference Submission21188 Authors

Keywords: large language model, differential privacy, synthetic data generation, reinforcement fine-tuning

TL;DR: We propose DP-RFT, a differentially private method to train an LLM to generate high-fidelity synthetic text via reinforcement fine-tuning.

Abstract:

Differentially private (DP) synthetic data generation plays a pivotal role in advancing applications of large language model (LLM) involving private user data.
Generating synthetic data that resemble eyes-off private corpora involves a difficult trade-off. On one hand, DP finetuning methods offers formal privacy guarantees, yet requires the data owner to expose the raw content for model training. However, methods that avoid direct exposure to private data are bounded by an off-the-shelf, un-finetuned model, whose outputs often lack domain fidelity. %To get the best of both worlds, the research question is
Can we train an LLM to generate high-quality synthetic text without providing it individual private examples?
In this work, we introduce Differentially Private Reinforcement Fine-Tuning (DP-RFT), an online reinforcement learning algorithm for synthetic data generation with LLMs. DP-RFT leverages DP-protected nearest-neighbor votes from an eyes-off private corpus as a reward signal for on-policy synthetic samples generated by an LLM. The LLM iteratively learns to generate synthetic data to maximize the expected DP votes through Proximal Policy Optimization (PPO). We evaluate DP-RFT for long-form and domain-specific synthetic data generation, such as news articles, meeting transcripts, and medical article abstracts. Our experiments show that DP-RFT significantly outperforms private evolution and DP fine-tuning methods, such as DP-SGD, in terms of both the fidelity and downstream utility of the generated synthetic data.

Primary Area: applications to computer vision, audio, language, and other modalities

Submission Number: 21188