# __UOSR (Unified Open-set Recognition) Training__

This is the official implementation of :
[Project](https://www.) **|** [Paper & Supp](https://arxiv.org/abs/)

[author name](https://), 
[author name](https://), 

_submit to International Conference on Learning Representations_ (**ICLR**), 2023.

## __Table of Contents__
1. [Overview](#overview)
1. [Installation](#installation)
1. [Datasets](#datasets)
1. [Testing](#testing)
1. [Training](#training)
1. [Model Zoo](#model-zoo)
1. [Citation](#citation)


## __Overview__

We deeply analyze the **U**nified **O**pen-**s**et **R**ecognition (**UOSR**) task under different training and evaluation settings. UOSR has been proposed to reject not only unknown samples but also known but wrongly classified samples. Specifically, we first evaluate the UOSR performance of existing OSR methods and demonstrate a significant finding that the uncertainty distribution of almost all existing methods designed for OSR is actually closer to the expectation of UOSR than OSR. Second, we analyze how two training settings of OSR (i.e.,**pre-training** and **outlier exposure**) effect the UOSR. Finally, we formulate the **Few-shot Unified Open-set Recognition** setting, where only one or five samples per unknown class are available during evaluation to help identify unknown samples. Our proposed FS-KNNS method for the few-shot UOSR achieved state-of-the-art performance under all settings.


<!-- ### Demo -->

## __Installation__

### __Requirements and Dependencies__
- Linux: Ubuntu 18.04 LTS
- GPU: GeForce RTX 3090

<!-- ### Installation Steps
sdfsdfsd -->


## __Datasets__

This repo uses standard image datasets, i.e., CIFAR-100 for closed set training, and TinuImagenet-resize and LSUN resize test sets as two different unknowns. Please refer to [ODIN](https://github.com/facebookresearch/odin) to download the out-of-distribution dataset. For outlier data, we choose the cleaned("debiased) image dataset [300K Random Images](https://people.eecs.berkeley.edu/~hendrycks/300K_random_images.npy). More details could be found in  [Outlier Exposure](https://github.com/hendrycks/outlier-exposure).

## __Testing__

To test our pre-trained models , you need to download a model file and unzip it in **log** dir. The pre-trained weights are in [Model Zoo](#model-zoo). After testing, you will get the uncertainty score result files which can be used in `../UOSR_eval` for UOSR performance evaluation. The following directory tree is for your reference to place the files.
```
work_dirs    
├── log
│    ├── SOFTMAX
│    │   └── best_bit.pth
│    │   └── bit.pth
│    │   └── train.log
│    ├── ... ... 
├── bit_pytorch
├── bit_hyperrule.py
├── data
└── ... ...
```
Evaluation for models trained from scratch:
```
cd bit_pytorch
# run the method SOFTMAX trained from scratch on GPU_ID=0
sh run_test_scratch.sh 0 SOFTMAX

```
Evaluation for models with imagenet-pretrained:
```
cd bit_pytorch
# run the method SOFTMAX on GPU_ID=0
sh run_test_pretrained.sh 0 SOFTMAX
```



## __Training__
Let's take the softmax method as an example
Train from scartch
```
cd bit_pytorch
# run the method baseline with save name SOFTMAX trained from scratch on GPU_ID=0 
sh run_train.sh 0 SOFTMAX baseline

```
Fine tune from pretrained model
```
cd bit_pytorch
# run the method baseline with save name PRETRAINED_SOFTMAX trained from scratch on GPU_ID=0
sh run_train_pretrained.sh 0 PRETRAINED_SOFTMAX baseline

```
Train from scartch with outlier data
```
cd bit_pytorch
# run the method OE with save name OE trained from scratch on GPU_ID=0 with outlier data batch size 128
bash run_oe_train.sh 0 OE OE 128

```
Fine tune with outlier data from pretrained model
```
cd bit_pytorch
# run the method OE with save name PRETRAINED_OE from pretrained model on GPU_ID=0 with outlier data batch size 128
bash run_oe_pretrained.sh 0 PRETRAINED_OE OE 128

```

## __Model Zoo__
Pre-trained weight are too large to be stored in thie folder. We will make them public soon. If you want to get the results of Table 2, 3, 5, 6, 7, and 8 in the manuscript, simply follow `../UOSR_eval` to directly print tables in your terminal based on the uncertainty score result files we provide.

## __Citation__

If you find the code useful in your research, please cite:

```
@inproceedings{
  year = "2023"
}

```

## __License__
See [Apache-2.0 License](https://github.com//LICENSE)
## __Acknowledgement__

This repo contains modified codes from:

- [Learning Confidence Estimates for Neural Networks](https://github.com/uoguelph-mlrg/confidence_estimation): for implementation of baseline method [LC ](https://arxiv.org/abs/1802.04865).
- [Out-of-Distribution Detector for Neural Networks](https://github.com/facebookresearch/odin): for implementation of baseline method [ODIN ](https://arxiv.org/abs/1706.02690).
- [Learning Placeholders for Open-Set Recognition](https://github.com/zhoudw-zdw/CVPR21-Proser): for implementation of baseline method [PROSER ](https://openaccess.thecvf.com/content/CVPR2021/papers/Zhou_Learning_Placeholders_for_Open-Set_Recognition_CVPR_2021_paper.pdf).
- [Deep Anomaly Detection with Outlier Exposure](https://github.com/hendrycks/outlier-exposure): for implementation of baseline method [OE ](https://arxiv.org/abs/1812.04606).


We sincerely thank the owners of all these great repos!