SAMPLING TO DISTILL: KNOWLEDGE TRANSFER FROM OPEN-WORLD DATA
Primary Area: general machine learning (i.e., none of the above)
Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
Keywords: Data-free, Knowledge Transfer
Submission Guidelines: I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2024/AuthorGuide.
Abstract: Data-Free Knowledge Distillation (DFKD) is a novel task that aims to train high-performance student models using only the teacher network without original training data.
Despite encouraging results, existing DFKD methods rely heavily on generation modules with high computational costs.
Meanwhile, they ignore the fact that the generated and original data exist domain shifts due to the lack of supervision information.
Moreover, knowledge is transferred through each example, ignoring the implicit relationship among multiple examples.
To this end, we propose a novel Open-world Data Sampling Distillation (ODSD) method without a redundant generation process.
First, we try to sample open-world data close to the original data's distribution by an adaptive sampling module.
Then, we introduce a low-noise representation to alleviate the domain shifts and build a structured relationship of multiple data examples to exploit data knowledge.
Extensive experiments on CIFAR-10, CIFAR-100, NYUv2, and ImageNet show that our ODSD method achieves state-of-the-art performance.
Especially, we improve 1.50\%-9.59\% accuracy on the ImageNet dataset compared with the existing results.
Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors' identity.
No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.
Submission Number: 3043
Loading