Development of Mixed Reality Robot Control System Based on HoloLens

Xuanmeng Sha, Zixi Jia, Weidong Sun, Yida Hao, Xingang Xiao, Hanlu Hu

Abstract: By using HoloLens, a wearable mixed reality device, and Unity3D, a 3D real-time driving engine, to simulate the robot kinematics and dynamics control in a mixed reality environment, we can test the performance of the control algorithm in virtual space before the physical simulation of the robot control algorithm. This paper first introduces the method of constructing a parameter-adjustable manipulator model with arbitrary degrees of freedom using Unity3D, and then introduces the method of using C# programming to implement the inverse kinematics simulation of the manipulator, and establishes a six-degree-of-freedom manipulator inverse kinematics control model to verify the feasibility of this method. Finally, we use the classic one-stage inverted pendulum model to verify the feasibility of robot dynamics control in Unity3D. This method changes the situation that the existing simulation method is difficult to interact with the real environment. Users can use the virtual robot model to verify the feasibility of the control scheme in complex environments.