Towards Fair Knowledge Distillation using Student Feedback

Abhinav Java, Surgan Jandial, Chirag Agarwal

Primary Area: societal considerations including fairness, safety, privacy

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.

Keywords: Fairness, Foundation Models, Bias-aware Distillation

Submission Guidelines: I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2024/AuthorGuide.

TL;DR: The first meta-learning framework for performing fair distillation

Abstract: With the advent of large-scale foundation models and their success in diverse fields, Knowledge Distillation (KD) techniques are increasingly used to deploy them to edge devices with limited memory and computation constraints. However, most KD works focus on improving the prediction performance of the student model distilled from large teacher models, and there is little to no work in studying the effect of distillation on key fairness properties, ensuring trustworthy distillation. In this work, we propose a fairness-driven distillation framework, BIRD (BIas-awaRe Distillation), which introduces a FAIRDISTILL operator to collect feedback from the student through a meta learning-based approach and selectively distill teacher knowledge. We demonstrate that BIRD can be augmented with different KD methods to increase the performance of a wide range of foundation models and convolu- tional neural networks after distillation. Extensive experiments across three fairness datasets show the efficacy of our framework over existing state-of-the-art KD meth- ods, opening up new directions to develop trustworthy distillation techniques.

Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors' identity.

No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.

Submission Number: 4352