Real-time Hand-object Occlusion for Augmented Reality Using Hand Segmentation and Depth Correction

Yuhui Wu, Yue Liu, Jiajun Wang

Abstract: Hand-object occlusion is crucial to enhance the realism of Aug-mented Reality, especially for egocentric hand-object interaction scenes. In this paper, a hand segmentation-based depth correction approach is proposed, which can help to realize real-time hand-object occlusion. We introduce a lightweight convolutional neural net-work to quickly obtain real hand segmentation mask. Based on the hand mask, different strategies are adopted to correct the depth data of hand and non-hand regions, which can implement hand-object occlusion and object-object occlusion simultaneously to deal with complex hand situations during interaction. The experimental re-sults demonstrate the feasibility of our approach presenting visually appealing occlusion effects.