Towards Visually Plausible ExplanationsDownload PDF

31 Jan 2021 (modified: 05 May 2023)ML Reproducibility Challenge 2020 Blind SubmissionReaders: Everyone
Keywords: Variational Autoencoders, Attention maps, Latent space disentanglements, Anomalies
Abstract: \section*{\centering Reproducibility Summary} \subsection*{Introduction} The goal of this assignment is to firstly reimplement the work from the paper \cite{liu2020towards} and secondly extend their work. Liu et al. develops a new technique which visually explains VAE by generating attention maps from the learned latent space. Then the paper applies the VAE into two applications: anomaly localisation and latent space disentanglement. This paper reimplements Liu's experiments and compares our results with the results from the paper under restricted computational resources. The acquired results will be analyzed and compared to the original paper. \subsection*{Scope of Reproducibility} The paper \cite{liu2020towards} claims that their research takes steps towards visually explaining generative models by using a new method visual attention maps conditioned on the latent space of a variational autoencoder (VAE). Furthermore, the attention maps can be used to demonstrate the localisation of anomalies in images. This localisation method for anomaly detection, resulted in state-of-the-art performance in the MVTec-AD dataset. Moreover, a new learning objective was formed: attention disentanglement loss. This resulted in better performance on the Dpsrites dataset compared to state-of-the-arts methods. \subsection*{Methodology} For the first part of the experiment the author's code was used, and for the second experiment, the proposed method from \citep{kim2019disentangling} was used with the addition of the disentanglement metric mentioned in the original paper. The total training lasted around six hours for the anomaly localisation and for the disentanglement it lasted approximately 40 to 80 hours. Slight hyperparameter tuning was needed for some of the datasets. \subsection*{Results} The results showed that for the most part the anomaly localisation claims seem to be reproducible (except for the UCSD pedestrians dataset). But the disentanglement was not reproducible. \subsection*{What was easy} The easiest reproducible parts for this experiment was the part for which the code and documentation was available. \subsection*{What was difficult} There was a wide range of obstacles, when reproducing this code. Much of the information was dependent on the interpretation of the reader. There is no explicit documentation about the implementation in the paper; hyper-parameters, experiment set-up and model descriptions are not present. Also, the full second part of the implementation was missing from the code base. And many equations or formulations were not sufficiently explained. \subsection*{Communication with original authors} There was little communication, only an opened issue on their Github repository, giving a solution to a known issue.
Paper Url: https://openreview.net/forum?id=CrORjXGxoNk&referrer=%5BML%20Reproducibility%20Challenge%202020%5D(%2Fgroup%3Fid%3DML_Reproducibility_Challenge%2F2020)
Supplementary Material: zip
4 Replies

Loading