Optical flow-based video completion in spherical image sequences

Binbin Xu, Sarthak Pathak, Hiromitsu Fujii, Atsushi Yamashita, Hajime Asama

Abstract: Spherical cameras are widely used for robot perception because of their full 360 degree fields of view. However, the robot body is always seen in the view, causing occlusions. In this paper, we propose a video completion method which is able to remove occlusions fast and recover the occluded background accurately. We first estimate motion in a 2D dense spherical optical flow field. Then we interpolate the motion in the occlusion regions by solving least square minimization problems using polynomial models. Based on the interpolated motion, we recover the occluded regions by tracing the optical flow trajectory to find the corresponding pixels in other frames and warping them back to fill in the occlusions. We also provide a simple, yet fast solution to effectively remove occlusions in all regions of the image by utilizing the continuity of field of view of spherical images. In experiments, quantitative methods are conducted to demonstrate the effectiveness and efficiency of our proposed method by comparing the results with those from state-of-the-art methods.