A Novel Performance Evaluation Methodology for Single-Target Trackers

Matej Kristan, Jiri Matas, Ales Leonardis, Tomas Vojir, Roman Pflugfelder, Gustavo Fernandez, Georg Nebehay, Fatih Porikli, Luka Cehovin

Abstract:

This paper addresses the problem of single-target tracker performance evaluation. We consider the performance
measures, the dataset and the evaluation system to be the most important components of tracker evaluation and propose
requirements for each of them. The requirements are the basis of a new evaluation methodology that aims at a simple and
easily interpretable tracker comparison. The ranking-based methodology addresses tracker equivalence in terms of statistical
significance and practical differences. A fully-annotated dataset with per-frame annotations with several visual attributes is
introduced. The diversity of its visual properties is maximized in a novel way by clustering a large number of videos according to
their visual attributes. This makes it the most sophistically constructed and annotated dataset to date. A multi-platform evaluation
system allowing easy integration of third-party trackers is presented as well. The proposed evaluation methodology was tested
on the VOT2014 challenge on the new dataset and 38 trackers, making it the largest benchmark to date. Most of the tested
trackers are indeed state-of-the-art since they outperform the standard baselines, resulting in a highly-challenging benchmark.
An exhaustive analysis of the dataset from the perspective of tracking difficulty is carried out. To facilitate tracker comparison a
new performance visualization technique is proposed.