[Re] Identifying Through Flows for Recovering Latent RepresentationsDownload PDF

31 Jan 2021 (modified: 05 May 2023)ML Reproducibility Challenge 2020 Blind SubmissionReaders: Everyone
Keywords: identifiability, normalising, flow, seperability, entanglement, disentenglement, principle, latent, representation
Abstract: Scope of Reproducibility In this study, we evaluate the paper ’Identifying Through Flows for Recovering Latent Representations’. Specifically, we evaluate the papers’ claimed practical advantages and effectiveness of their proposed method iFlow over previous methods, namely iVAE. Methodology First, we reproduce the obtained MCC scores for both iFlow and iVAE using the original code-base. To place these results into context, we also evaluate two baseline Flow models. Furthermore, we discuss the proposed method’s usability, and apply it on a different Flow model, which is trained on the Half Moon dataset to analyse the learned latent representation. With this, we assess the benefit of using the proposed method over regular Flow. It takes around 20 minutes for an iFlow model, and 75 seconds for an iVAE model to train and evaluate conform to the original implementation and dataset on an RTX 2080 Ti GPU with 11GB of VRAM. Additionally, the iFlow network with planar-flow Flow model takes around 7 minutes to train on the same hardware. Finally, the two Flow models on the half moon datasets are trained on an AMD 3900X CPU with 32GB of DDR4 RAM, each taking roughly 3 minutes. Results Our results are within 2.5% of the values presented by the paper, verifying the authors’ claim of iFlow’s theoretical advancements over iVAE. However, when compared to the baseline Flow models, iFlow only shows up to 10% improvement in MCC scores, compared to a 45% improvement over iVAE. Furthermore, when analysing the learned latent representation for the Half Moon dataset iFlow does learn a more robust latent representation compared to Flow, and unlike Flow, is sometimes able to reach principled disentanglement, partly verifying the paper’s claim of iFlow’s practical advantages and effectiveness. What was easy The original code implementation was not difficult to setup and run specifically for the iFlow model. The code provided the proper run script for training and evaluating iFlow. Furthermore, implementing the proposed identifiability method to different flow models is not difficult - the authors provide a clear derivation of the objective function. Finally, in the code, a different use of the activation for the natural parameters is suggested, which we found to be straightforward to implement. What was difficult The code-base lacked documentation, thus besides running the default iFlow setup, running different models such as iVAE was quite challenging. In general, understanding the code itself, particularly the code used to generate the dataset, was not straightforward. No code was offered to save the results and construct the figures from the paper. Finally, despite the supposed support for using planar-flow instead of the default cubic spline-flow in the code base, training iFlow with planar-flow was not trivial. This was due to both an incorrect initialization of the planar-flow model, where it called the wrong class, and an incorrect return statement. Communication with original authors There has been no communication with the authors.
Paper Url: https://openreview.net/forum?id=QxBBSsd_j9b
Supplementary Material: zip
3 Replies

Loading