A Multi-party Conversational Social Robot Using LLMs

Angus Addlesee, Neeraj Cherakara, Nivan Nelson, Daniel Hernández García, Nancie Gunson, Weronika Sieinska, Marta Romeo, Christian Dondrup, Oliver Lemon

Abstract: In this paper, we describe our setting and the architecture of our LLM-based dialogue system embodied in a social robot and able to have multi-party conversations. Each component is detailed, and a video of the full system is available with the appropriate components highlighted in real-time. Our system decides when it should take its turn, generates human-like clarification requests when the patient pauses mid-utterance, answers in-domain questions (grounding to the in-prompt knowledge), and responds appropriately to out-of-domain requests (like generating jokes or quizzes).