A flexible technique to select objects via convolutional neural network in VR space

Huiyu Li, Linwei Fan

Abstract: Most studies on the selection techniques of projection-based VR systems are dependent on users wearing complex or expensive input devices, however there are lack of more convenient selection techniques. In this paper, we propose a flexible 3D selection technique in a large display projection-based virtual environment. Herein, we present a body tracking method using convolutional neural network (CNN) to estimate 3D skeletons of multi-users, and propose a region-based selection method to effectively select virtual objects using only the tracked fingertips of multi-users. Additionally, a multi-user merge method is introduced to enable users’ actions and perception to realign when multiple users observe a single stereoscopic display. By comparing with state-of-the-art CNN-based pose estimation methods, the proposed CNN-based body tracking method enables considerable estimation accuracy with the guarantee of real-time performance. In addition, we evaluate our selection technique against three prevalent selection techniques and test the performance of our selection technique in a multi-user scenario. The results show that our selection technique significantly increases the efficiency and effectiveness, and is of comparable stability to support multi-user interaction.