Reproducibility of "Pixel-wise Anomaly Detection in Complex Driving Scenes" for ML Reproducibility Challenge 2021

Anonymous

Keywords: Autonomous Driving, Anomaly Detection, Anomaly Segmentation

TL;DR: This paper is a reproducibility report of "Pixel-wise Anomaly Detection in Complex Driving Scenes"

Abstract:

Reproducibility Summary: The following paper is a reproducibility report for Pixel-wise Anomaly Detection in Complex Driving Scenes published in CVPR 2021 as part of the ML Reproducibility Challenge 2021. We reproduced the results quantitatively, performed ablation studies, re-implemented the model in PyTorch Lightning and integrated WandB.

Scope of Reproducibility: Our efforts are focused on validating the authors’ proposed anomaly segmentation framework, which employs the latest re-synthesis approaches and extends them to incorporate the advantages of uncertainty estimation methods. This proposed model outperforms existing re-synthesis techniques by a significant margin on the task of anomaly segmentation on the Fishyscapes dataset.

Methodology: We initially re-implemented the dissimilarity module in PyTorch Lightning using the authors’ publicly available source code. PyTorch Lightning increases the readability, reproducibility, and robustness of the code. It also provides distributed training. We used pre-trained weights for image segmentation, image reconstruction and trained the dissimilarity model on the Cityscapes dataset. We trained all the models on a single P-100 GPU offered by Kaggle for over 850 training hours.

Results: Overall, our results back the original paper’s claims. Our model outperforms the original study on a few metrics but slightly falls behind on others on the benchmarked Fishyscapes dataset.

What was easy: The paper was well-written and easy to understand. The provided open-source code is well-structured and modular. Having pre-trained weights available for standard segmentation and reconstruction models reduced computational load.

What was difficult: Even with modular code available, re-implementing the code in PyTorch Lightning proved more challenging than expected. Our experiments were limited by the model’s computational constraints, with an average training duration of 25 hours per model and Kaggle only providing 9 hours of continuous training time. The data generation method is not specified in the original repository; We have created a single script in our repository for the same.

Communication with original authors: Authors were contacted via email to help clarify queries about the code, dataset generation, and discrepancies in the results. Our final report received a positive review from the authors.

Paper Url: https://arxiv.org/pdf/2103.05445.pdf

Paper Venue: CVPR 2021