Knowledge Distillation as Decontamination? Revisiting the “Data Laundering” Concern

ICLR 2026 Conference Submission18278 Authors

Keywords: Knowledge Distillation, Data Contamination, Benchmark Integrity, Data Decontamination

Abstract: Concerns have been raised that knowledge distillation may transfer test-set knowledge from a contaminated teacher to a clean student—a “data laundering” effect that potentially threatens evaluation integrity. In this paper, we assess the severity of this phenomenon. If these concerns regarding data laundering are minor, then distillation could be used to mitigate risks of direct data exposure. Across eight benchmarks, we find that substantial laundering is the exception rather than the rule: unlike the large performance gains from direct contamination, any accuracy inflation from laundering is consistently smaller and statistically insignificant in all but two cases. More broadly, we find that the two phenomena are weakly correlated, suggesting that laundering is not simply a diluted form of contamination but a distinct effect that arises primarily when benchmarks exhibit large train–test distribution gaps. Motivated by this, we conduct controlled experiments that systematically enlarge the train–test distance on two benchmarks where laundering was initially negligible, and observe that laundering becomes more significant as the gap widens. Taken together, our results indicate that knowledge distillation, despite rare benchmark-specific residues, can be expected to function as an effective decontamination technique that largely mitigates test-data leakage.

Primary Area: transfer learning, meta learning, and lifelong learning

Submission Number: 18278