Foreground Depth Estimation for Semi-automatic 2D-to-3D Video Conversion

Wen-Nung Lie, Yi-Kai Chen, Jui-Chiu Chiang

Abstract: Depth map estimation is important in 2D to 3D video conversion. Normally, the background part is static or changes slowly, while the foreground part might change substantially between consecutive frames. A good strategy is that depths for the foreground and the background parts are estimated separately and then combined together to form the final depth map. In this paper, we propose, for non-key frames, an algorithm of automatic foreground depth propagation from key frames where the foreground part is segmented and depth-assigned manually with some supporting computer tools. For each non-key frame, the foreground region is segmented independently based on the graph-cut and GMM (Gaussian Mixture Model) algorithms. The superpixel algorithm is then applied to the foreground area only for partitioning it into homogeneous patches. To propagate/compensate the foreground depths from key frames, superpixel matching (based on color component and foreground labels) is performed between each non-key frame and its reference frame, with the background parts removed. We then refine the foreground depths by using bilateral filtering. Experiments show that compared to conventional algorithms of block matching, optical flow, and superpixel, our method is advantageous of resisting large foreground motion and erroneous matching caused by background interference (similar colors). In overall, our algorithm improves the resulting foreground depth map significantly.