A Coarse-to-Fine Framework for Automatic Video Unscreen

Anyi Rao, Linning Xu, Zhizhong Li, Qingqiu Huang, Zhanghui Kuang, Wayne Zhang, Dahua Lin

Abstract:

Video unscreen, a technique to extract foreground
from given videos, has been playing an important role in today’s
video production pipeline. Existing systems developed for this
purpose which mainly rely on video segmentation or video
matting, either suffer from quality deficiencies or require tedious
manual annotations. In this work, we aim to develop a fully
automatic video unscreen framework that is able to obtain
high-quality foreground extraction without the need of human
intervention in a controlled environment. Our framework adopts
a coarse-to-fine strategy, where the obtained background estimate
given an initial mask prediction in turn helps the refinement
of the mask by the alpha composition equation. We conducted
experiments on two datasets, 1) the Adobe’s Synthetic-Composite
dataset, and 2) DramaStudio, our newly collected large-scale
green screen video matting dataset, exhibiting the controlled
environments. The results show that the proposed framework
outperforms existing algorithms and commercial software, both
quantitatively and qualitatively. We also demonstrate its utility
in person replacement in videos, which can further support a
variety of video editing applications.