# Knowledge distillation for memorized data: Soft labels can leak held out information

Code accompanying the submission.

## Usage

All experiments use the basic code available at `mod_addition_experiment.py`, `logistic_regression_experiment.py` or `experiment.py`.

For reproducibility we record most experiments to a wandb database, along with the random seeds that were used to create them. Upon publication, we will make this database accessible publicly.
Until then, one can use the experiment (`.py`) files to re-run them, and the plotting notebooks (`.ipynb`) to create the plots for the figures in the corresponding names of the files.

For the visual examples `fig_1` and `fig_7` the seeds are configured in the plotting file directly.
For logistic regression in `fig_3` we save and provide the results in the repository.

> Disclaimer: In order to load the results from the correct wandb repository, the repository of the ipynb needs to be the same as the one in the py file

## Resources

All experiments can be run both on cpu or gpu - for multinomial logistic regression cpu may be faster than a gpu.

The most computational intensive were the phase diagrams Fig. 4a) with 13 compute days, and Fig. 6a) with roughly 10 compute days on a GPU. In both cases though, we ran the full pipeline for parallel experiments on a single machine with an NVIDIA RTX A5000 within roughly two days.

Since many of the experiments are run for different seeds to obtain error bars, running the experiments once is roughly a factor 5 faster than running all. Since experiments do not require large resources they can be parallelized easily.