Point-VOS: Pointing Up Video Object Segmentation

Sabarinath Mahadevan

Abstract:

Current state-of-the-art Video Object Segmentation
(VOS) methods rely on dense per-object mask annota-
tions both during training and testing. This requires time-
consuming and costly video annotation mechanisms. We
propose a novel Point-VOS task with a spatio-temporally
sparse point-wise annotation scheme that substantially re-
duces the annotation effort. We apply our annotation
scheme to two large-scale video datasets with text descrip-
tions and annotate over 19M points across 133K objects in
32K videos. Based on our annotations, we propose a new
Point-VOS benchmark, and a corresponding point-based
training mechanism, which we use to establish strong base-
line results. We show that existing VOS methods can easily
be adapted to leverage our point annotations during train-
ing, and can achieve results close to the fully-supervised
performance when trained on pseudo-masks generated from
these points. In addition, we show that our data can be
used to improve models that connect vision and language,
by evaluating it on the Video Narrative Grounding (VNG)
task.