[RE] Zero-Shot Knowledge Transfer via Adversarial Belief Matching

Alexandros Ferles, Alexander Nöu, Leonidas Valavanis

Abstract: We reproduce the work in Zero-shot Knowledge Transfer via Adversarial Belief Matching, which describes a novel approach for knowledge transfer. A teacher network trained on real samples distills knowledge to a student network that is trained solely on pseudo data extracted from a generator network, with the student trying to mimic the teacher’s outputs on these datapoints. To this end, we additionally re-implement Wide Residual Networks which are used as the main framework for both teacher and student networks and train them from scratch on CIFAR10 and SVHN. We compare the results of the proposed method with a few-shot knowledge distillation attention transfer setting implemented and trained from scratch. We suggest an approach for further exploitation of the learnt mechanics of the generator network in the zero-shot setting, which operates on top of the main method, and briefly discuss the benefits and drawbacks of this approach. Our code can be found publicly available in https://github.com/AlexandrosFerles/NIPS_2019_Reproducibilty_Challenge_Zero-shot_Knowledge_Transfer_via_Adversarial_Belief_Matching

Track: Baseline

NeurIPS Paper Id: https://openreview.net/forum?id=B1x00VHgIH