Your ViT is Secretly Also a Video Segmentation Model

Narges Norouzi, Idil Esen Zulfikar, Niccolò Cavagnero, Tommie Kerssies, Bastian Leibe, Gijs Dubbelman, Daan de Geus

Keywords: Vision Transformer, Efficient Video Segmentation, Encoder-only Mask Transformer, Video Segmentation

Abstract: Existing online video segmentation models typically combine a per-frame segmenter with complex specialized tracking modules. While effective, these modules introduce significant architectural complexity and computational overhead. Recent studies suggest that plain Vision Transformer (ViT) encoders, when scaled with sufficient capacity and large-scale pre-training, can conduct accurate image segmentation without requiring such specialized components. Motivated by this observation, we propose the Video Encoder-only Mask Transformer (VidEoMT), a simple encoder-only video segmentation model that eliminates the need for dedicated tracking modules. To enable temporal modeling in an encoder-only ViT, VidEoMT introduces a lightweight query fusion mechanism that merges queries from the previous frame with temporally-agnostic learned queries, enabling information propagation across frames while preserving adaptability to new content. As a result, VidEoMT attains the benefits of a tracker without added complexity and achieves competitive accuracy, while being 5-10x faster, running at up to 160 FPS with a ViT-L backbone. Code will be made public upon acceptance.

Primary Area: applications to computer vision, audio, language, and other modalities

Submission Number: 7949