[Re] Learning Intra-Batch Connections for Deep Metric Learning

Anonymous

Abstract:

Scope of Reproducibility

In this work we reproduce the paper by Seidenschwarz et al. (2021). They introduce a method that aims to more successfully capture the global structure of embedding space in the task of metric learning by constructing graphs of mini-batches and applying Message Passing Network (MPN) with dot product self-attention on them. They argue that using MPN alleviates the need for specially designed loss functions and that their method can be trained with cross-entropy loss while still achieving state-of-the-art performance on image retrieval and clustering tasks. We reproduce their experiments in order to verify this main claim and further explore two additional claims.

Methodology

We reproduce the original experiments using slightly modified authors’ code. Firstly, we reevaluate the trained models provided by the authors. Secondly, we replicate their results by training our own models on CUB-200-2021, Cars196, Stanford Online Products and In-Shop datasets. Furthermore, we perform additional experiments to uncover how the attention mechanism evolves during training and whether that agrees with authors’ secondary claims. Running all the experiments took approximately 47 GPU hours on Nvidia TITAN X GPU.

Results

Recall@K scores reported in the paper and the results obtained by reevaluating authors’ models completely agree in almost every case. The same can be observed for the models we trained from scratch with differences no larger than 0.5pp on all datasets except for Cars196. There, the discrepancies are around 1pp which is also relatively low. The experimental results strongly support the main claim of the paper.

What was easy

The authors’ code was very well organized. The instructions were sufficient to create an environment and download the relevant data. Once the repository was successfully set up, training and evaluating the models was very straightforward.

What was difficult

Initially, the code contained a few minor bugs that had to be fixed in order to run it. Because the repository is very large and contains a lot of code with many possible ways to configure the models but not a lot of comments and documentation, it took us more time than expected to unravel the meaning of all the possible parameters and configurations.

Communication with original authors

We found the original paper in combination with the authors’ code sufficient to infer all the necessary information and replicate the main findings. Therefore, we have not contacted the authors.

Paper Url: https://icml.cc/Conferences/2021/Schedule?showEvent=10238

Paper Venue: ICML 2021