Rethinking Knowledge Transfer in Learning Using Privileged Information

TMLR Paper3499 Authors

Abstract: In supervised machine learning, privileged information (PI) is information that is unavailable at inference, but is accessible during training time. Research on learning using privileged information (LUPI) aims to transfer the knowledge captured in PI onto a model that can perform inference without PI. It seems that this extra bit of information ought to make the resulting model better. However, finding conclusive theoretical or empirical evidence that supports the ability to transfer knowledge using PI has been challenging. In this paper, we critically examine the assumptions underlying existing theoretical analyses and argue that there is little theoretical justification for when LUPI should work. We analyze two main LUPI methods - generalized distillation and marginalization with weight sharing - and reveal that apparent improvements in empirical risk may not directly result from PI. Instead, these improvements often stem from dataset anomalies or modifications in model design misguidedly attributed to PI. Our experiments for a wide variety of application domains further demonstrate that state-of-the-art LUPI approaches fail to effectively transfer knowledge from PI. Thus, we advocate for practitioners to exercise caution when working with PI to avoid unintended inductive biases.

Submission Length: Regular submission (no more than 12 pages of main content)

Changes Since Last Submission:

- We have refined the abstract to better reflect the scope of our study, explicitly emphasizing the two LUPI methods examined;
- We have adjusted our conclusions to ensure they align with the specific methods studied and avoid overly broad claims;
- We have expanded our discussion to acknowledge cases where privileged information may provide benefits beyond standard performance metrics.

Code: https://github.com/danilprov/rethinking_lupi

Assigned Action Editor: ~Novi_Quadrianto1

Submission Number: 3499