Indoor Dialog Agent in Mixed Reality

KyuHwon Park, Young-Seob Jeong

Abstract: We aimed at implementing an indoor dialog agent, namely PhoenixBot, working in a mixed-reality environment. The agent occupies a certain position in a real-world space, and interacts with other nearby human.We developed a server that maintains information of agents and smartphone users, where the information includes current indoor position and direction. We also developed an Android application as a client, which collects real-time data from various sensors such as gyroscope, accelerometer, step detector, WiFi, magnetic field, and gravity sensor. The step detector values and WiFi signals are used to estimate the current location of the user, and the other remaining sensors are used to compute the user direction. The client application displays the real-world scene covered with some virtual objects (e.g., agent, board), as depicted in Fig. 1, where the cartoon character at the center is the dialog agent. In order to compute the right position to display the components, the client keeps consistent information of the virtual objects with the server. That is, if a user moves to other position, then it will be reported to the server and will be disclosed to the other agents and users. The dialog agent works in the way similar to other dialog agents, as it has a pipeline of modules such as Natural Language Understanding (NLU), Dialog Management (DM), and Natural Language Generation (NLG). The agent currently supports four domains: weather, campus, transport, and bible. The agent speaks only Korean for now, but will be portable to other langauges if a dataset for the target language is prepared. You can see the video clip at https://youtu.be/U2FA-XxVPvM