PianoSyncAR: Enhancing Piano Learning through Visualizing Synchronized Hand Pose Discrepancies in Augmented Reality

Ruofan Liu, Erwin Wu, Chen-Chieh Liao, Hayato Nishioka, Shinichi Furuya, Hideki Koike

Abstract: Motor skill acquisition involves learning from spatiotemporal discrepancies between target and self-generated motions. However, in dexterous skills with numerous degrees of freedom, understanding and correcting these motor errors are challenging. This issue becomes crucial for experienced individuals who seek for mastering and sophisticating their skills, where even subtle errors need to be minimized. To enable efficient optimization of body posture in piano learning, we present PianoSyncAR, an augmented reality system that superimposes the time-varying complex hand postures of a teacher over the hand of a learner. Through a user study with 12 pianists, we demonstrate several advantages of the proposed system over conventional tablet-screen, which implicate the potential of AR training as a complementary tool for video-based skill learning in piano playing.