Imitate Your Own Refinement: Knowledge Distillation Sheds Light on Efficient Image-to-Image Translation

Linfeng Zhang, Jiebo Song, Runpei Dong, Xin Chen, Ke Ding, Kaisheng Ma

Keywords: Knowledge Distillation, Image-to-Image Translation, Self-Distillation, Model Compression

TL;DR: We propose a novel knowledge distillation method for efficient image-to-image translation by replacing the teacher network with a refining network.

Abstract: The excellent performance of the state-of-the-art Generative Adversarial Networks (GANs) is always accompanied by enormous parameters and computations, making them unaffordable on resource-limited mobile devices. As an efficient model compression technique, knowledge distillation has been proposed to transfer the knowledge from a cumbersome teacher to a lightweight student. Following its success on classification, some recent works have applied knowledge distillation to GAN-based image-to-image translation but lead to unsatisfactory performance. In this paper, to tackle this challenge, we propose a novel knowledge distillation framework named IYOR (Imitate Your Own Refinement), which consists of the following two techniques. Firstly, since image-to-image translation is an ill-posed problem, knowledge distillation on image-to-image translation may force the student to learn the average results between multiple correct answers and thus harm student performance. To address this problem, we propose to replace the teacher network in knowledge distillation with a refining network, which is trained to refine the images generated by the student to make them more realistic. During the training period, the refining network and the student are trained simultaneously, and the student is trained to imitate the refined results in a knowledge distillation manner. Secondly, instead of only distilling the knowledge in the generated images, we propose SIFT KD, which firstly extracts the distinctive and scale-invariant features of the generated images with Scale-invariant feature transform (SIFT), and then distills them from the refining network to the student. Extensive experimental results demonstrate the effectiveness of our method on five datasets with nine previous knowledge distillation methods. Our codes are released in the supplementary material and will be released on GitHub.

Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.

No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics

Submission Guidelines: Yes

Please Choose The Closest Area That Your Submission Falls Into: Deep Learning and representational learning