Hang Out with the Language Assistant

Jin-hwan Oh, Sudhakar Sah, Jihoon Kim, Yoori Kim, Jeonghwa Lee, Wooseung Lee, Myeongsoo Shin, Jaeyon Hwang, Seongwon Kim

Abstract: AI assistants have found their place in households but most of the existing assistants use single modal interaction. We present a language assistant for kids called Hola (Hang out with the Language Assistant) which is a true multimodal assistant. Hola is a small mobile robot based assistant capable of understanding the objects around it and responding to questions about objects that it can see. Hola is also able to adjust the camera position and its own position to make an extra attempt to understand the object using robot control mechanism. The technology behind it uses a combination of natural language understanding, object detection, and hand pose detection. In addition, Hola also supports reading book in the form of storytelling for kids using OCR. Children can ask a question about any word that they do not understand and Hola can retrieve the information from the internet and tells the meaning, other details of the word. After reading the book or a page, the robot asks the child based on the words used in the book to confirm the child’s understanding of the book.