MultiMediate : Multimodal Behaviour Analysis for Artificial Mediation

Philipp Müller, Marius Funk, Daksitha Withanage Don, Elisabeth Andre, Victor Oei, Andreas Bulling, Michal Balazia, Francois Bremond

Keywords: multimodal behaviour analysis, engagement, generalisation

Abstract: The MultiMediate challenge is a multi-year endeavour to lay the foundations for socially capable artificial mediators that can support human interactions. In the its first years, MultiMediate has focused on solving basic social behaviour sensing tasks including eye contact detection, backchannel detection, and bodily behaviour recognition. More recently, the challenge focused on the development of algorithms that can estimate the degree by which a human is engaged in a social interaction - a complex phenomenon that is strongly influenced by cultural norms and the nature of the social situation. By providing a diverse set of training and testing datasets, MultiMediate has facilitated the development of generalisable engagement estimation approaches. In MultiMediate ’26, the diversity of training and evaluation data is enriched further by including the PInSoRo dataset of child-child and child-robot interactions annotated with both social and task engagement. As such, MultiMediate ’26 poses the challenging task of creating engagement estimation approaches that are able to transfer between different social situations, languages, age groups, and notions of engagement.

Email Sharing: We authorize the sharing of all author emails with Program Chairs.

Data Release: We authorize the release of our submission and author names to the public in the event of acceptance.

Submission Number: 24