Leveraging RGB-Pressure for Whole-body Human-to-Humanoid Motion Imitation

Yi Lu, Shenghao Ren, Qiu Shen, Xun Cao

Abstract:

Whole-body motion imitation has gained wide attention in recent years as it can enhance the locomotive capabilities of humanoid robots.
In this task, non-intrusive human motion capturing with RGB cameras is commonly used for its low-cost, efficiency, portability and user-friendliness. However, RGB based methods always faces the problem of depth ambiguity, leading to inaccurate and unstable imitation.
Accordingly, we propose to introduce pressure sensor into the non-intrusive humanoid motion imitation system for two considerations: first, pressure can be used to estimate the contact relationship and interaction force between human and the ground, which play a key role in the balancing and stabilizing motion; second, pressure can be measured in the manner of almost non-intrusive approach, which can keep the experience of human demonstrator. In this paper, we establish a RGB-Pressure (RGB-P) based humanoid imitation system, achieving accurate and stable end-to-end mapping from human body models to robot control parameters. Specifically, we use RGB camera to capture human posture and pressure insoles to measure the underfoot pressure during the movements of human demonstrator. Then, a constraint relationship between pressure and pose is studied to refine the estimated pose according to the support modes and balance mechanism, thereby enhancing consistency between human and robot motions. Experimental results demonstrate that fusing RGB and pressure can enhance overall robot motion execution performance by improving stability while maintaining imitation similarity.

Primary Subject Area: [Content] Multimodal Fusion

Secondary Subject Area: [Experience] Interactions and Quality of Experience

Relevance To Conference: Humaniod robot motion imitation based on motion capturing is an emerging an important topic to enlarge the applications of multimedia. It may promote the development of multimedia technologies including image processing, image analysing, multimodal sensing, feature fusion, action understanding.

Submission Number: 2909