TTVOS: Lightweight Video Object Segmentation with Adaptive Template Attention Module and Temporal Consistency Loss

Hyojin Park, Ganesh Venkatesh, Nojun Kwak

Keywords: Semi-supervised video segmentation, video tracking, lightweight segmentation

TL;DR: we propose an adaptive template matching method and a novel temporal consistency loss for Semi-supervised video object segmentation

Abstract:

Semi-supervised video object segmentation (semi-VOS) is widely used in many applications.
This task is tracking class-agnostic objects by a given segmentation mask.
For doing this, various approaches have been developed based on online-learning, memory networks, and optical flow.
These methods show high accuracy but are hard to be utilized in real-world applications due to slow inference time and tremendous complexity.
To resolve this problem, template matching methods are devised for fast processing speed, sacrificing lots of performance.
We introduce a novel semi-VOS model based on a temple matching method and a novel temporal consistency loss to reduce the performance gap from heavy models while expediting inference time a lot.
Our temple matching method consists of short-term and long-term matching.
The short-term matching enhances target object localization, while long-term matching improves fine details and handles object shape-changing through the newly proposed adaptive template attention module.
However, the long-term matching causes error-propagation due to the inflow of the past estimated results when updating the template.
To mitigate this problem, we also propose a temporal consistency loss for better temporal coherence between neighboring frames by adopting the concept of a transition matrix.
Our model obtains 79.5% J&F score at the speed of 73.8 FPS on the DAVIS16 benchmark.