Partitioning variability in animal behavioral videos using semi-supervised variational autoencoders

Matthew R. Whiteway, Dan Biderman, Yoni Friedman, Mario Dipoppa, Estefany Kelly Buchanan, Anqi Wu, John Zhou, Niccolò Bonacchi, Nathaniel J. Miska, Jean-Paul Noel, Erica Rodriguez, Michael Schartner, Karolina Socha, Anne E. Urai, C. Daniel Salzman, International Brain Laboratory, John P. Cunningham, Liam Paninski

Abstract: Author summary The quantification of animal behavior is a crucial step towards understanding how neural activity produces coordinated movements, and how those movements are affected by genes, drugs, and environmental manipulations. In recent years video cameras have become an inexpensive and ubiquitous way to monitor animal behavior across many species and experimental paradigms. Here we propose a new computer vision algorithm that extracts a succinct summary of an animal’s pose on each frame. This summary contains information about a predetermined set of body parts of interest (such as joints on a limb), as well as information about previously unidentified aspects of the animal’s pose. Experimenters can thus track body parts they think are relevant to their experiment, and allow the algorithm to discover new dimensions of behavior that might also be important for downstream analyses. We demonstrate this algorithm on videos from four different experimental setups, and show how these new dimensions of behavior can aid in downstream behavioral and neural analyses.