Exploring Depth Information for Object Segmentation and Detection

Tyng-Luh Liu, Kai-Yueh Chang, Shang-Hong Lai

Abstract: We propose a new framework for performing object segmentation and detection simultaneously. Our method leverages with an MRF graphical model that comprises two kinds of nodes and two types of labels for inference. Specifically, we decompose an image into super pixels and generate segment proposals from each super pixel. The super pixels are then duplicated to form the two types of nodes. For each segmentation node, the model is to predict the object class label, while it is to decide the label corresponding to the best segment proposal selection at each detection node. The former is clearly a segmentation problem and the latter a detection problem. We link the two tasks by establishing a unified energy function that has a joint energy term accounting for the compatibility of the segmentation and detection labelings. Marginalizing by fixing either type of variables, the energy function can be switched into the one specifically for detection or segmentation. This property enables an alternating procedure to conveniently obtain the optimal labelings. To better explain the geometry about the objects and the scene, we use the depth information so that 3-D distances between super pixels are available in computing each energy term. Experimental results on a dataset with depth information are provided to support the effectiveness of our method.