# OODRobustBench
A benchmark of out-of-distribution adversarial robustness

## Prepare datasets

To prepare datasets, first download them from the corresponding links (you can only download CIFAR10 datasets if not plan to run experiments for ImageNet):

CIFAR10:
* CIFAR10.1: https://github.com/modestyachts/CIFAR-10.1
* CIFAR10.2: https://github.com/modestyachts/cifar-10.2
* CINIC-10: https://datashare.ed.ac.uk/handle/10283/3192

ImageNet:
* ImageNet-v2: https://huggingface.co/datasets/vaishaal/ImageNetV2/tree/main
* ImageNet-A: https://people.eecs.berkeley.edu/~hendrycks/imagenet-a.tar
* ImageNet-R: https://people.eecs.berkeley.edu/~hendrycks/imagenet-r.tar
* ObjectNet: https://objectnet.dev/downloads/objectnet-1.0.zip

Next, name the directory of each dataset as below (some of them should be already named properly):

* CIFAR10: cifar-10-batches-py
* CIFAR10.1: cifar-10.1
* CIFAR10.2: cifar-10.2
* CINIC-10: CINIC-10
* CIFAR10-R: cifar-10-r
* CIFAR10-C: CIFAR-10-C
* ImageNet: imagenet
* ImageNet-C: ImageNet-C
* ImageNet-v2: imagenetv2-matched-frequency-format-val
* ImageNet-A: imagenet-a
* ImageNet-R: imagenet-r
* ObjectNet: objectnet

Data preparation steps for each dataset:

- CIFAR10.1: Move `npy` files in datasets/ to root directory of CIFAR10.1.

## Dependencies

```
pip install -r requirements.txt
```

## Evaluation

evaluate CIFAR10 models from RobustBench with all corruptions and natural shifts for 10k samples (`-n 10000`):

```bash
python oodar/eval.py --data_dir ./data --threat-model Linf --corruption-models corruptions --natural-shifts all -n 10000 --model_name Andriushchenko2020Understanding
```

you need to set the `data_dir` as the root directory of all datasets, and change the `model_name` to the RobustBench ID of the model to be evaluated.

## Custom Models

Model name is a bit long and starts with `custom_`. It contains the hyperparameter choices. For example, `custom_convmixer_trades_trades_seed0_bs512_lr0.1_wd0.0001_sgd_50ep_eps0.5_beta0.1`,

The weights are hosted on Zenodo and is downloaded automatically when a model is called. Download speed from Zenodo server can be poor sometimes so if you know you want to use all the models, you can download all the weights at once with `zenodo_get` and put them at the right location:

```bash
pip install zenodo_get
cd $MODEL_PATH  # mkdir if needed
zenodo_get $DEPOSIT_ID
```

where `$DEPOSIT_ID` and `$MODEL_PATH` are the Zenodo deposit ID and the associated model path. Each deposit has a maximum size of 40 GB and contains a group of models denoted by the path. See the list of `$DEPOSIT_ID: $MODEL_PATH` below:

* `8285099`: `cifar10/L2`.

Please install the extra packages in `requirements.txt`. See `oodar.models.custom_models.utils._MODEL_DATA` for the list of available models.
