A Cautionary Tale on the Evaluation of Differentially Private In-Context Learning

Anjun Hu, Jiyang Guan, Philip Torr, Francesco Pinto

Track: Technical

Keywords: Differential Privacy, AI Safety, In-Context Learning, LLM Evaluation

TL;DR: Language models' pre-training knowledge and biases can significantly impact the efficacy and generalizability of DP-ICL methods.

Abstract:

In-context learning (ICL) has emerged as a powerful paradigm enabling Large Language Models (LLMs) to perform new tasks by prompting them with few training examples, eliminating the need for fine-tuning. Given its potential to adapt and personalize the model's behaviour using private user data, recent studies have introduced techniques for ICL that satisfy Differential Privacy guarantees (DP ICL). Existing DP ICL approaches claim to attain such guarantees while maintaining negligible utility degradations when adapting the models to perform new tasks.
In this paper, we present preliminary empirical evidence suggesting that these claims may hold only for tasks aligned with the model's pre-training knowledge and biases. We do so by showing the performance of DP ICL significantly degrades with respect to the non-private counterpart in scenarios that introduce tasks and distribution shifts that challenge the model's prior knowledge.
To mitigate the risk of overly optimistic evaluations of DP ICL, we invite the community to consider our sanity checks to attain a more accurate understanding of its capabilities and limitations.

Submission Number: 9