MITIGATING BIAS IN DATASET DISTILLATION

Justin Cui, Ruochen Wang, Yuanhao Xiong, Cho-Jui Hsieh

Keywords: dataset distillation, dataset condensation

TL;DR: mitigating bias in the synthetic dataset generated by dataset distillation/condensation process

Abstract:

Dataset distillation (DD) has emerged as a technique for compressing large datasets into smaller synthetic counterparts, facilitating downstream training tasks. In this paper, we study the impact of bias within the original dataset on the performance of dataset distillation. With a comprehensive empirical evaluation on datasets with color, corruption and background biases, we found that color and background biases in the original dataset will be amplified through the distillation process, resulting in a notable decline in the performance of models trained on the synthetic set, while corruption bias is suppressed through the distillation process.
To reduce bias amplification in dataset distillation,
we introduce a simple yet highly effective approach based on a sample reweighting scheme utilizing kernel density estimation.
Empirical results on multiple datasets demonstrated the effectiveness of the proposed method.
Notably, on CMNIST with 5\% bias-conflict ratio and IPC 50, our method achieves 91.5\% test accuracy compared to 23.8\% from vanilla DM, boosting the performance by 67.7\%, whereas applying state-of-the-art debiasing method on the same synthetic set only achieves 53.7\%.
Our findings highlight the importance of addressing biases in dataset distillation and provide a promising avenue to mitigate bias amplification in the process.

Primary Area: representation learning for computer vision, audio, language, and other modalities

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.

Submission Guidelines: I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2024/AuthorGuide.

Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors' identity.

No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.

Submission Number: 3835