
# Cosine Similarity KD (CSKD) #
**COSINE SIMILARITY KNOWLEDGE DISTILLATION FOR INDIVIDUAL CLASS INFORMATION TRANSFER**

## Installation

Environments:

- Python 3.8
- PyTorch 1.7.0

Install the package:

```
pip install -r requirements.txt
python setup.py develop
```

## Getting started

1. Evaluation

- You can see "cskd_eval.sh"


  ```bash 
  # CSKD

  ## teacher: resnet32x4, student: shuffleV1
  python tools/eval.py -m ShuffleV1 -c best_results/res32x4_shufv1/student_best

  ## teacher: wrn_40_2, student: shuffleV1
  python tools/eval.py -m ShuffleV1 -c best_results/wrn40_2_shufv1/student_best
  ```


2. Training

- Unable to include the weight of the teacher model due to attachment limitations (only ResNet56 is included). Therefore, the weights of the teacher models should be created before implementing our CSKD. (Refer to "teacher.sh")

  ```bash
  # teacher models
  python tools/train.py --cfg configs/cifar100/teacher/wrn_40_2.yaml
  python tools/train.py --cfg configs/cifar100/teacher/res32x4.yaml
  python tools/train.py --cfg configs/cifar100/teacher/res110.yaml
  python tools/train.py --cfg configs/cifar100/teacher/vgg13.yaml
  python tools/train.py --cfg configs/cifar100/teacher/res50.yaml

  # move "student_best" into "download_ckpts/cifar_teachers/"
  mv output/cifar100_teacher/teacher,wrn_40_2/student_best download_ckpts/cifar_teachers/wrn_40_2_vanilla/ckpt_epoch_240.pth
  mv output/cifar100_teacher/teacher,res32x4/student_best download_ckpts/cifar_teachers/resnet32x4_vanilla/ckpt_epoch_240.pth
  mv output/cifar100_teacher/teacher,res110/student_best download_ckpts/cifar_teachers/resnet110_vanilla/ckpt_epoch_240.pth
  mv output/cifar100_teacher/teacher,vgg13/student_best download_ckpts/cifar_teachers/vgg13_vanilla/ckpt_epoch_240.pth
  mv output/cifar100_teacher/teacher,res50/student_best download_ckpts/cifar_teachers/ResNet50_vanilla/ckpt_epoch_240.pth
  ```

- If the teacher models are prepared, you can train the student models using our CSKD, as demonstrated in the "cskd_scripts.sh".

  ```bash
  # cskd
  python tools/train_ours.py --cfg configs/cifar100/cskd/res56_res20.yaml
  ```

