Reproducibility Report: D3S - A Discriminative Single Shot Segmentation Tracker

Anonymous

Keywords: neural network, visual object tracking, reproducibility

Abstract:

Reproducibility
The original paper describes the architecture of the D3S neural network and evaluates its performance in the task of visual object tracking and video segmentation tasks. In our reproducibility study, we focused on training and evaluation of D3S for visual object tracking tasks due to limited time.

Methodology
Our work is based on code provided by the authors of the original paper. The training code was reorganized and partially re-implemented. As a result, our version consists of only the most necessary code (the original code consists of other experiments not presented in the paper). For model evaluation, we use the pytracking framework following the authors of the original article. We used NVIDIA Tesla V100 GPU with CUDA 9.2 and pytorch 1.7.1 for model training and validation. The time it took to train the model was 16 hours.

Results
The difference of the reproduced model quality metrics does not exceed 3%. These differences do not change the position of D3S relative to other architectures in comparison. It is found that the speed of model evaluation (FPS) differs significantly for different datasets, whereas the original paper provided a single estimate of a speed. At the same time, the obtained values are lower than the ones given in the article. The reason for the differences may be the various hardware configurations of the computers used for the experiments.

What was easy
The open-source code of the authors was very helpful. Also, the evaluation pipeline in visual object training is not trivial, and the authors of the original code use the pytracking framework for this task. It is significantly reduced the complexity of our work.

What was difficult
We had a few problems due to incompatibilities between the versions of pytorch and CUDA used in the original code and required to work with our hardware. In addition, it is not clear from the original paper how metrics were calculated from the raw output (bounding boxes): by toolkits supplied with datasets or somehow else.

Communication with original authors
We did not communicate with the authors at all, except to use their publicly available source code.

Paper Url: https://openreview.net/forum?id=6N0v-QkkLD&noteId=7kb86zoxZWa