Using Theory of Mind to Assess Users' Sense of Agency in Social Chatbots

Evelien Heyselaar, Tibor Bosse

Abstract: The technological advancements in the field of chatbot research is booming. Despite this, it is still difficult to assess which social characteristics a chatbot needs to have for the user to interact with it as if it had a mind of its own. Review studies have highlighted that the main cause is the low number of research papers dedicated to this question, and the lack of a consistent protocol within the papers that do address it. In the current paper, we suggest the use of a Theory of Mind task to measure the implicit social behaviour users exhibit towards a text-based chatbot. We present preliminary findings suggesting that participants adapt towards this basic chatbot significantly more than when they conduct the task alone (p < .017). This task is quick to administer and does not require a second chatbot for comparison, making it an efficient universal task. With it, a database could be built with scores of all existing chatbots, allowing fast and efficient meta-analyses to discover which characteristics make the chatbot appear more ‘human’.