## Required Packages:

- torch (1.10.1+cu102)
- torchvision (0.11.2+cu102)
- numpy
- matplotlib
- torchattacks
- path
- git+https://github.com/fra31/auto-attack


## Initial setup

Download all files in https://rebrand.ly/roget-iclr and put "CIFAR10_models_LATEST" in src/models/defense/ and all other content in src/


## Reproducing results from the paper

We found that torchattacks and https://github.com/fra31/auto-attack give conflict on some systems, causing auto-attack to give error. We suggest creating separate virtual environments for white box and black box evaluation. The details of creating these environments are listed below.

Create virtual environment for white box evaluation in ROGET_supp/

```
$ virtualenv white_box_eval
$ source white_box_eval/bin/activate
$ pip install torch==1.10.1+cu102 torchvision==0.11.2+cu102 -f https://download.pytorch.org/whl/torch_stable.html
$ pip install matplotlib
$ pip install cleverhans
$ pip install git+https://github.com/fra31/auto-attack
$ pip install path
```

Create virtual environment for black box evaluation

```
$ virtualenv black_box_eval
$ source black_box_eval/bin/activate
$ pip install torch==1.10.1+cu102 torchvision==0.11.2+cu102 -f https://download.pytorch.org/whl/torch_stable.html
$ pip install matplotlib
$ pip install cleverhans
$ pip install torchattacks
$ pip install path
```

Now make sure you are in src directory. All commands should be run from src.


# Uncertainty based subset selection with default hyperparameters (Table 1) 

For e.g. to get numbers for Auto Attack on CIFAR10:
```
$ source ../white_box_eval/bin/activate 
$ python -m clean_model_expt -d cifar10
$ python evaluate_on_test_suggested.py -d cifar10 -a auto-att
```

# Uncertainty based subset selection with worst case hyperparameter setup (Table 2)

For e.g. to get numbers for MIFGSM on FMNIST:
```
$ source ../black_box_eval/bin/activate 
$ python clean_model_expt.py -d fmnist
$ python evaluate_baselines_on_val_worst_case.py -d fmnist
$ python evaluate_on_test_worst_case.py -d fmnist -a mifgsm
```

# Evaluation on class based subset selection strategy (Table 3)

For e.g. to get numbers for Auto Attack attack on "Airplane" class of CIFAR10:
```
$ source ../white_box_eval/bin/activate 
$ python exec/generate_label_subsets.py -d cifar10 -c 0
$ python evaluate_baselines_on_val_worst_case.py -d cifar10
$ python evaluate_on_class_attack.py -d cifar10 -a auto-att -c 0
```

# Impact of revealing the oracle subset selection strategy during validation (Table 4)

For e.g. to get numbers for PGD attack on CIFAR10:
```
$ source ../white_box_eval/bin/activate 
$ python clean_model_expt.py -d cifar10
$ python clean_model_expt.py -d cifar10 --val
$ python evaluate_baselines_on_val_oracle.py -d cifar10
$ python evaluate_on_test_oracle.py -d cifar10 -a pgd
```

# Variation of A vs |S_latent|

For e.g. to get numbers for MIFGSM attack on CIFAR10:
```
$ source ../black_box_eval/bin/activate 
$ python clean_model_expt_frac.py -d cifar10 -f 0.05
$ python clean_model_expt_frac.py -d cifar10 -f 0.1
$ python clean_model_expt_frac.py -d cifar10 -f 0.15
$ python clean_model_expt_frac.py -d cifar10 -f 0.2
$ python clean_model_expt_frac.py -d cifar10 -f 0.25
$ python evaluate_baselines_on_val_worst_case.py -d cifar10
$ python evaluate_on_various_fractions.py -d cifar10 -a mifgsm -f 0.05
$ python evaluate_on_various_fractions.py -d cifar10 -a mifgsm -f 0.1
$ python evaluate_on_various_fractions.py -d cifar10 -a mifgsm -f 0.15
$ python evaluate_on_various_fractions.py -d cifar10 -a mifgsm -f 0.2
$ python evaluate_on_various_fractions.py -d cifar10 -a mifgsm -f 0.25
$ python exec/generate_frac_plot.py -d cifar10 -a mifgsm
```
You can find the plot at src/testset_results_frac/plot_cifar10_mifgsm.png


### Training our models

To train our model using the default values on CIFAR10 dataset, run the following commands (from either environment). The first one trains with a*{\phi} = PGD and the second one trains with a*{\phi}=AdvGAN.

```
$ python exec/train.py --attack_model_type=pgd
$ python exec/train.py --attack_model_type=advgan
```

The models will be created in saved_models/cifar10/ inside a timestamped directory. Models are saved at every iteration. Copy the saved model and replace it with "model-final" in any Roget_PGD or Roget_AdvGAN directory (as the case may be) to repeat any evaluation above on the trained model.

To train our model using the default values on FMNIST dataset, run the following commands (from either environment). The first one trains with a*{\phi} = PGD and the second one trains with a*{\phi}=AdvGAN.

```
$ python exec/train_fmnist.py --attack_model_type=pgd
$ python exec/train_fmnist.py --attack_model_type=advgan
```

The models will be created in saved_models/fmnist/ inside a timestamped directory. Models are saved at every iteration. Copy the saved model and replace it with "model-final" in any Roget_PGD or Roget_AdvGAN directory (as the case may be) to repeat any evaluation above on the trained model.


### Evaluate models

To evaluate the models on test set:
For e.g to evaluate our method for hyperparameter 1.0, with auto-att on CIFAR10:
```
$ python load_and_eval_vecs.py -d cifar10 -a auto-att -m Ours_1.0
```
For validation set,
```
$ python load_and_eval_vecs_validation.py -d cifar10 -a auto-att -m Ours_1.0
```

This requires a results/cifar10/Our_1.0/model-final to be present.
the results directory should be at the same directory level as the supplementary folder.
i.e. from src, ../../results/cifar10/Our_1.0/model-final should be present
