PATEin: A Privacy-Preserving Framework for Knowledge Integration via Adaptive Teacher Selection in C-LLMs

ICLR 2026 Conference Submission16372 Authors

Keywords: Commercial large language models (C-LLMs)，In-context learning（ICL），Privacy-Preserving，Private aggregation of teacher ensembles (PATE)，Knowledge integration

Abstract: In-context learning (ICL) enables task adaptation without modifying model parameters, making it well-suited for commercial large language models (C-LLMs) with closed-source constraints. However, ICL prompts often contain sensitive information, raising significant privacy concerns. Most existing privacy-preserving methods for ICL require access to model parameters, making them incompatible with C-LLMs. Recent methods based on teacher ensembles with differentially private aggregation have shown promise but face two fundamental challenges: ensemble inconsistency and limited knowledge integration. We propose PATEin, a novel privacy-preserving knowledge transfer framework that dynamically selects the optimal individual teacher model for labeling, thereby mitigating the loss of individual knowledge. Furthermore, it introduces a supervised teacher strategy that selectively incorporates high-consistency voting, effectively integrating individual and ensemble knowledge. Experiments on various C-LLMs (e.g., GPT-3.5-turbo, GPT-4o-mini, Claude-3.5-haiku, DeepSeek-v3) demonstrate that PATEin significantly improves labeling accuracy, reduces computational overhead, and consistently outperforms existing baseline methods.

Primary Area: alignment, fairness, safety, privacy, and societal considerations

Submission Number: 16372