# Knowledge Distillation under Distribution Shift

**benchmarks 12 knowledge distillation methods in PyTorch, including:**

- (KD) - Distilling the Knowledge in a Neural Network
- (FitNet) - Fitnets: hints for thin deep nets
- (AT) - Paying More Attention to Attention: Improving the Performance of Convolutional Neural Networks via Attention Transfer
- (FT) - Paraphrasing Complex Network: Network Compression via Factor Transfer
- (AB) - Knowledge Transfer via Distillation of Activation Boundaries Formed by Hidden Neurons
- (NST) - Like what you like: knowledge distill via neuron selectivity transfer
- (SP) - Similarity-Preserving Knowledge Distillation
- (PKT) - Probabilistic Knowledge Transfer for deep representation learning
- (VID) - Variational Information Distillation for Knowledge Transfer
- (RKD) - Relational Knowledge Distillation
- (CC) - Correlation Congruence for Knowledge Distillation
- (CRD) - Contrastive Representation Distillation

  

**benchmarks 12 data manipulation methods in PyTorch, including:**

- Data Augmentation
  - (ImageNet baseline)
  - (AutoAugment) - [Autoaugment: Learning augmentation strategies from data](https://scholar.google.com/scholar_url?url=http://openaccess.thecvf.com/content_CVPR_2019/html/Cubuk_AutoAugment_Learning_Augmentation_Strategies_From_Data_CVPR_2019_paper.html&hl=zh-CN&sa=T&oi=gsb&ct=res&cd=0&d=8586846733647481474&ei=RywIZeD9FquJ6rQP87mMqAI&scisig=AFWwaeZv1QAKL2BSZndnFSNoCgUf)
  - (RandAugment) - [Randaugment: Practical automated data augmentation with a reduced search space](https://scholar.google.com/scholar_url?url=http://openaccess.thecvf.com/content_CVPRW_2020/html/w40/Cubuk_Randaugment_Practical_Automated_Data_Augmentation_With_a_Reduced_Search_Space_CVPRW_2020_paper.html&hl=zh-CN&sa=T&oi=gsb&ct=res&cd=0&d=10903590128857500448&ei=USwIZaSLDoG7yATum7PwBg&scisig=AFWwaeb1D0Tlcr-nn5gQIyBKVU2S)
  - (Mixup) - [**mixup**: Beyond empirical risk minimization](https://scholar.google.com/scholar_url?url=https://arxiv.org/abs/1710.09412&hl=zh-CN&sa=T&oi=gsb&ct=res&cd=0&d=12669856454801555406&ei=NCwIZfv1IoLyyATpwqGYDA&scisig=AFWwaeYYx7R2aCkkVvdTRUjqhK-w)
  - (CutMix) - [**Cutmix**: Regularization strategy to train strong classifiers with localizable features](https://scholar.google.com/scholar_url?url=http://openaccess.thecvf.com/content_ICCV_2019/html/Yun_CutMix_Regularization_Strategy_to_Train_Strong_Classifiers_With_Localizable_Features_ICCV_2019_paper.html&hl=zh-CN&sa=T&oi=gsb&ct=res&cd=0&d=15618183235315733915&ei=WiwIZYulE432yATflamIBA&scisig=AFWwaea00JaGyqOMGlJHw_hBa3m3)
  - (DomainMix) - [**Domainmix**: Learning generalizable person re-identification without human annotations](https://scholar.google.com/scholar_url?url=https://arxiv.org/abs/2011.11953&hl=zh-CN&sa=T&oi=gsb&ct=res&cd=0&d=7576833949867176269&ei=YiwIZcb-FpHHywTmq5JA&scisig=AFWwaeZPR4eUNfB7oJhS-92kVfyb)
  - (MixStyle) - [Domain generalization with **mixstyle**](https://scholar.google.com/scholar_url?url=https://arxiv.org/abs/2104.02008&hl=zh-CN&sa=T&oi=gsb&ct=res&cd=0&d=4489212027125038279&ei=aywIZbypA5eI6rQPlKy8mAo&scisig=AFWwaeZPbmJyvOo4_vQ1tIhZ11vl)
  - Gaussian noise
- Data Pruning
  - (Random Prune) - pruning the data with random selection
  - (EL2N) - [Deep learning on a data diet: Finding important examples early in training](https://scholar.google.com/scholar_url?url=https://proceedings.neurips.cc/paper_files/paper/2021/hash/ac56f8fe9eea3e4a365f29f0f1957c55-Abstract.html&hl=zh-CN&sa=T&oi=gsb&ct=res&cd=0&d=6692350500928309521&ei=eCwIZaepMKyR6rQPk7mgiAI&scisig=AFWwaeZ2fu3keoPPQe2umMGfqtKH)
  - (GraNd) - [Deep learning on a data diet: Finding important examples early in training](https://scholar.google.com/scholar_url?url=https://proceedings.neurips.cc/paper_files/paper/2021/hash/ac56f8fe9eea3e4a365f29f0f1957c55-Abstract.html&hl=zh-CN&sa=T&oi=gsb&ct=res&cd=0&d=6692350500928309521&ei=eCwIZaepMKyR6rQPk7mgiAI&scisig=AFWwaeZ2fu3keoPPQe2umMGfqtKH)

#  Datasets

Our code supports the following dataset:

- Office-Home
- PACS
- DomainNet
- ColorMNIST
- CelebA-Blond

# Acknowledgment

Great thanks to [DomainBed](https://github.com/facebookresearch/DomainBed)，[RepDistiller](https://github.com/HobbitLong/RepDistiller)，[Dassl.pytorch](https://github.com/KaiyangZhou/Dassl.pytorch)，**[transferlearning/DeepDG](https://github.com/jindongwang/transferlearning/tree/60d89070549701c4a75fe5b1ac625264820d5ca8/code/DeepDG)**. 