TTVOS: Lightweight Video Object Segmentation with Adaptive Template Attention Module and Temporal Consistency LossDownload PDF

Published: 07 Feb 2021, Last Modified: 03 Nov 2024tinyML 2021 RegularReaders: Everyone
Keywords: Semi-supervised video segmentation, video tracking, lightweight segmentation
TL;DR: we propose an adaptive template matching method and a novel temporal consistency loss for Semi-supervised video object segmentation
Abstract: Semi-supervised video object segmentation (semi-VOS) is widely used in many applications. This task is tracking class-agnostic objects by a given segmentation mask. For doing this, various approaches have been developed based on online-learning, memory networks, and optical flow. These methods show high accuracy but are hard to be utilized in real-world applications due to slow inference time and tremendous complexity. To resolve this problem, template matching methods are devised for fast processing speed, sacrificing lots of performance. We introduce a novel semi-VOS model based on a temple matching method and a novel temporal consistency loss to reduce the performance gap from heavy models while expediting inference time a lot. Our temple matching method consists of short-term and long-term matching. The short-term matching enhances target object localization, while long-term matching improves fine details and handles object shape-changing through the newly proposed adaptive template attention module. However, the long-term matching causes error-propagation due to the inflow of the past estimated results when updating the template. To mitigate this problem, we also propose a temporal consistency loss for better temporal coherence between neighboring frames by adopting the concept of a transition matrix. Our model obtains 79.5% J&F score at the speed of 73.8 FPS on the DAVIS16 benchmark.
Community Implementations: [![CatalyzeX](/images/catalyzex_icon.svg) 4 code implementations](https://www.catalyzex.com/paper/ttvos-lightweight-video-object-segmentation/code)
4 Replies

Loading