<h1 align="center">A simple Training-Free Method for Rejection Option</h1>

## Install requirements

On a fresh virtual environment, please run:

```bash
pip install -r requirements.txt
```

## Train the models

To train the classification models with CE, please run the following script:

```shell
python3 -m src.utils.vision.train.basic \
    --model_name ${model_name} \
    --seed ${seed} \
    --config_file src/utils/vision/train/configs/cifar10.json
```

where `${model_name}` is the name of the model to be trained, and `${seed}` is the random seed. Choose between:

```python
model_names = [
    "densenet121_cifar10",
    "densenet121_cifar100",
    "densenet121_svhn",
    "resnet34_cifar10",
    "resnet34_cifar100",
    "resnet34_svhn",
    "vgg16_cifar10",
    "vgg16_cifar100",
    "vgg16_svhn",
]
seeds = [1,2,3,4,5]
```

## Reproduce results

To reproduce our results, please run the following:

```shell
python3 -m src.selection.benchmarks.abstention \
    --model_name ${model_name} \
    --score_name gini \
    --seed ${seed} --batch_size 256
```

## Results

- CIFAR-10
  
<img src="images/results_cifar10.png" align="center"/>

- CIFAR-100

<img src="images/results_cifar100.png" align="center"/>

- SVHN

<img src="images/results_svhn.png" align="center"/>

## Baselines

To run the baselines, please check the following repositories:

- SelectiveNet: `https://github.com/geifmany/selectivenet`; 
- ConfidNet: `https://github.com/valeoai/ConfidNet`; 
- DeepGamblers: `https://github.com/Z-T-WANG/NIPS2019DeepGamblers`.
