A Commonsense Knowledge-based Object Retrieval Approach for Virtual Reality

Haiyan Jiang, Dongdong Weng, Xiaonuo Dongye, Nan Zhang, Luo Le

Abstract: Out-of-reach object retrieval is an important task in many applications in virtual reality. Hand gestures have been widely studied for object retrieval. However, the one-to-one mapping metaphor of the gesture would cause ambiguity problems and memory burdens for the retrieval of plenty of objects. Therefore, we proposed a grasping gesture-based object retrieval approach for out-of-reach objects based on a graphical model called And-Or graphs (AOG), leveraging the scene-object occurrence, object co-occurrence, and human grasp commonsense knowledge. This approach enables users to acquire objects by using natural grasping gestures according to their experience of grasping physical objects. Importantly, users could perform the same grasping gesture for different virtual objects and perform different grasping gestures for one virtual object in the virtual environment.