Uncertainty Estimation Using a Single Deep Deterministic Neural Network - ML Reproducibility Challenge 2020

Yash Vardhan Sharma, Rohan Mallick

Keywords: Uncertainty, aleatoric, epistemic, gradient penalty, RBF network

Abstract:

Scope of Reproducibility

The investigated paper claims RBF network when trained with BCE loss along with two-sided gradient penalty outperforms deep ensemble in the task of out of distribution(OoD) detection along with competitive accuracy to softmax based models. The Paper claims to outperform AUROC on Fashion-MNIST vs MNIST and CIFAR-10 vs SVHN. The proposed algorithm is reported to detect both aleatoric and epistemic uncertainty as OoD. Authors mention the need for a formal way to distinguish between the two kinds of uncertainty and pose it as an interesting future research avenue.
The scope of this report is to reproduce the results related to OoD detection presented in the paper. Along with the reproduction of results, we propose an extension for explicit detection of aleatoric and epistemic uncertainty as intended by the authors.

Methodology

The author's training code is available on GitHub. Additionally, we have made available all the experimentation codes in the form of notebooks. We provide all the results and analysis on the models described in the paper and trained on NVIDIA Tesla T4 GPU.

Results

Our reproduction supports the claim of paper, we are able to replicate exact trends as described in the paper, all results are within 1$\%$ of the value reported. Also, results of our proposed extension and its analysis are quite encouraging.

Overall our reproduction supports the claims of the paper, we can replicate trends and plots as described in the paper. Most of the results are within 1$\%$ of the value reported. Notably, AUROC(M) of DUQ in OoD detection of Fashion-MNIST vs MNIST is off by 1.5$\%$ and we got a different optimal value for gradient penalty weight ($\lambda$) in it.
Also, the results of our proposed extension and its analysis are encouraging. Our proposed extension provided an increase of 1.8$\%$ in AUROC(M) in Fashion-MNIST vs MNIST and provided explicit control over the aleatoric and epistemic uncertainty.

What was easy

The proposed approach is quite simple and elegant. The availability of the author's code made the implementation of various experiments easy.

What was difficult

Understanding of proposed approach requires advanced knowledge of calculus related to the Lipschitz constant and its role to quantify the upper bound of the sensitivity of any function.

Communication with original authors

We discussed our report with the authors, they find our analysis on aleatoric uncertainty interesting and appreciate our proposed extension and its encouraging results.

Paper Url: https://openreview.net/forum?id=IrTQ3lPwzcN&referrer=%5BML%20Reproducibility%20Challenge%202020%5D(%2Fgroup%3Fid%3DML_Reproducibility_Challenge%2F2020)