Prediction of Hand Kinematics in Grasping with Mamba-Based Graph Convolutional Networks

Weiting Peng, Jing Tang, Zeyu Gong, Bo Tao

Abstract: Prediction of hand kinematics in grasping contributes to accurate and efficient robotic grasping. In existing methods, the topology graph of hands only focuses on local constraints of hand structure, neglecting global coordination by the brain. Also, hand kinematics can be viewed as multivariate time series collected by sensors, so utilizing advanced neural networks can improve prediction accuracy. Therefore, we propose a virtual node-optimized topology graph and design Mamba-based graph convolutional networks (MambaGCN). Our topology graph achieves global coordination using virtual nodes, optimizing the aggregation of multivariate time series. We introduce the advanced Mamba architecture to enhance the modeling of multivariate time series. This study conducts on predicting hand kinematics in grasping using multivariate time series collected by sensors, which can serve as a pretext task for denoising and guiding robotic grasping. On the publicly available HANDdata dataset, results indicate that the proposed method outperforms three advanced models in terms of accuracy. Ablation studies validate the effectiveness of the virtual node-optimized topology graph and the improved Mamba architecture. The framework of MambaGCN is open-sourced at https://github.com/White-oranges/MambaGCN.