Thoughtful faces: Using facial features to infer naturalistic cognitive processing across species

Alejandro Tlaie, Katharine Shapcott, Muad Abd El Hay, Pierre-Antoine Ferracci, Robert Taylor, Iuliia Glukhova, Martha N Havenith, Marieke Schölvinck

Abstract: Understanding how internally driven cognitive processes (COPs) give rise to behavior in different species is one of the key challenges in systems neuroscience. Traditionally, neuroscience has aimed to isolate individual COPs by simplifying sensory stimuli and behavioral readouts. Yet in natural behavior, several COPs can co-occur, often with overlapping neural representations. The dynamics of such superimposed COPs are yet to be understood, as well as the extent to which they generalize across species. To tackle this question, we developed a naturalistic Virtual Reality setup in which two species (mice and macaques) engage in the same visual foraging task. To infer underlying COPs in a data-driven manner, we employed a recently developed approach that extends Hidden Markov Models (HMMs) by allowing their probabilities to vary according to Generalized Linear Models (GLMs). We exploited the richness of a wide range of facial features extracted from video recordings to train a GLM-HMM. By doing so, we identified, on a single-trial basis, a set of internal states that predicted whether animals could proficiently perform the task. In both species, facial features just before the animal’s decision were most predictive of trial outcome, and could be distilled into a low number of internal states. While internal states mapped onto specific trial outcomes in both species, these states did not rely on the same raw facial features in monkeys and mice. Transitions between internal states were frequent, precluding convergence onto one dominant state. However, in mice slower state transitions improved model performance, while this association was reversed for macaques. This suggests that in mice, behaviorally relevant internal states evolve on a slower time scale than in monkeys. With this framework, we were able to flexibly and agnostically track the dynamics of several ongoing COPs, identifying general principles of naturalistic cognitive processing across species.