# Replay Scheduling for Continual Learning using Monte Carlo tree search  

This repository includes the code in PyTorch for our paper "Learn the Time to Learn: Replay Scheduling for Continual Learning".

## Installation

* Create conda environment from file:
```
conda env create -f environment.yml
conda activate rs_mcts
```
* The following structure is in the main directory:
```
/data               : Directory with datasets not provided by torchvision
/src                : Main directory where all scripts are placed
/src/configs        : Directory with config files
/src/mcts           : Directory where MCTS code is placed
/src/trainer        : Directory where trainers for all methods are placed 
/src/training       : Directory where networks and helper code for training is placed
```

## Running Experiments
Run all experiemnts for all datasets within ```src/``` directory. The config files for each experiment contains the hyperparameters that we used in the paper. All experiments are run for the 5 seeds used in the paper, i.e. SEED=1,2,3,4,5. 

* Split MNIST (5 tasks, 2 classes/task). Runtime: 5 seeds*1h20min
```
python run_mcts.py ./configs/rs_mcts/MNIST.yaml
```
* Split FashionMNIST (5 tasks, 2 classes/task) Runtime: 5 seeds*1h45min
```
python run_mcts.py ./configs/rs_mcts/FashionMNIST.yaml
```
* Split notMNIST (5 tasks, 2 classes/task) Runtime: 5 seeds*1h45min (Has to be preprocessed! See information below)
```
python run_mcts.py ./configs/rs_mcts/notMNIST.yaml
```
* Permuted MNIST (10 tasks, 10 classes/task) Runtime: 5 seeds*8.7h 
```
python run_mcts.py ./configs/rs_mcts/PermutedMNIST.yaml
```
* Split CIFAR-100 (20 tasks, 5 classes/task) Runtime: 5 seeds*7.3h
```
python run_mcts.py ./configs/rs_mcts/CIFAR100.yaml
```
* Split miniImagenet (20 tasks, 5 classes/task) Runtime: 5 seeds*7.8h (Has to be downloaded and preprocessed! See information below)
```
python run_mcts.py ./configs/rs_mcts/miniImagenet.yaml
```
**Be aware of the long runtimes**

The results from each experiment are saved in directory ```experiment/DATASET/rs_mcts/${SAMPLE_SELECTION}/M${MEMORY_SIZE}```.

The memory selection method can be changed by changing the parameter **sample_selection** in the config files, for example:
```
sample_selection: uniform
sample_selection: kmeans_features
sample_selection: kcenter_features
sample_selection: icarl
```

Experiments for the **ETS baselines** are executed similarly for all datasets:
```
python run_ets_baseline.py ./configs/ets/${DATASET}.yaml
```

Experiments for the **Breadth-First Search (BFS)** are executed similarly (only for Split MNIST, FashionMNIST and notMNIST):
```
python run_bfs.py ./configs/bfs/${DATASET}.yaml
```



## Running Tiny Memory Experiments (Sec 4.5)
Experiments in the tiny memory setting with RS-MCTS are executed similarly as above for all datasets:  
```
python run_mcts.py ./configs/tiny_memory_experiment/rs_mcts/${DATASET}.yaml
```
**Be aware of the long runtimes again**

Experiments for the baselines are executed similarly 
```
python run_baseline.py ./configs/tiny_memory_experiment/baselines/${BASELINE}/${DATASET}.yaml
```
where the argument ${BASELINE}="agem" or "er_ring" or "er_uniform".

The results from each experiment are saved in directory ```experiment_tiny_memory/${DATASET}/${METHOD}/M${MEMORY_SIZE}```.

## Running Single Task Replay Memory Experiment (Sec. 1 and Fig.1)
The experiment with the single task replay memory with task 1 data on Split MNIST can be run by executing: 
```
python run_exp_single_task_memory.py ./configs/single_task_replay_MNIST.yaml
```

The runtime for this experiment is arund 20 mins. Results from the experiment are saved in directory ```experiment/single_task_replay_MNIST```.


## Additional Setup Steps for Split notMNIST
A zip-folder of notMNIST dataset has been downloaded from the [Adversarial CL repository](https://github.com/facebookresearch/Adversarial-Continual-Learning). The zip-folder is stored in the directory ./data.

Some images could not be opened by the dataloaders, so we need to filter these images before running experiments. 

Execute the following command in the root directory:
```
python data/filter_notMNIST.py
```
The notMNIST dataset is then stored in ```src/datasets/notMNIST``` with subdirectories ```train/``` and ```test/``` 
with training and test images respectively. 

## Additional Setup Steps for Split miniImagenet
miniImagenet dataset was downloaded from [this Google drive repo with jpg files](https://drive.google.com/open?id=137M9jEv8nw0agovbUiEN_fPl_waJ2jIj), where we found the link in [this repo](https://github.com/yaoyao-liu/mini-imagenet-tools#about-mini-ImageNet). 

Download the directories ./train, ./val, and ./test from the Google drive. We need to split all classes into a training and a test set. Keep their subdirs in one directory stored in ./data directory and run the following command in the root directory:
```
python data/process_folder_miniimagenet.py
```
The splitted miniImagenet dataset is then stored in ```src/datasets/miniImagenet``` with subdirectories ```train/``` and ```test/``` 
with training and test images respectively.
