Origami Sensei: Mixed Reality AI-Assistant for Creative Tasks Using Hands

Qiyu Chen, Richa Mishra, Dina El-Zanfaly, Kris Kitani

Abstract: Learning creative tasks that involve using hands and the body, like knitting and music playing, can be challenging for beginners. We introduce Origami Sensei, a mixed-reality system that assists beginners in creating origami by providing step-by-step instructions in real-time. Origami Sensei enables users to create their own origami pieces from a library of pre-existing designs. It uses computer vision algorithms to identify the user’s current origami step. It then provides them with real-time visual projections and verbal feedback to assist them in completing the next steps. The current setup consists of a tablet, a reflective mirror and a small projector. This research approach opens possibilities for introducing mixed-reality systems for learning hands-on skills in other fields such as welding, sculpting and even surgical operations. Future steps for this project include developing multimodal instructions and conducting user studies.