Versatile multiple object tracking in sparse 2D/3D videos via deformable image registration

James Ryu, Amin Nejatbakhsh, Mahdi Torkashvand, Sahana Gangadharan, Maedeh Seyedolmohadesin, Jinmahn Kim, Liam Paninski, Vivek Venkatachalam

Abstract: Author summary While deep learning with convolutional neural networks has been successfully applied to many multiple object tracking problems, these advances do not immediately generalize to videos of fluorescence reported dynamics in living tissue, where the combination of sparse global distributions and locally dense, homogeneous peaks present a challenging instance of a multiple object tracking problem. Imaging such sparse fluorescent signals is a standard tool for observing neuronal activity in genetically engineered animals, and performing imaging in naturally behaving animals to place that activity in the context of behavior only increases the difficulty of the problem. Thus, this step is typically a significant bottleneck in efforts to understand the relationship between neuronal activity and naturalistic behavior. We build upon recent advances in spatial transformers and differentiable grid sampling to develop a new registration-based approach: ZephIR, a semi-supervised multiple object tracking algorithm with a novel cost function that can incorporate a diverse set of spatio-temporal constraints that can change dynamically during optimization. Local registration of image features enables tracking of keypoints even in sparse imaging conditions, such as fluorescent cellular data, while a spring network incorporates a flexible motion model of the neighboring keypoints without the need for a highly specialized skeletal model. Feature detection can help fine-tune tracking results to match a nearby detected feature in the image or even recover good tracking accuracy in cases where registration clearly fails to produce good gradients. In addition, any amount of new manual labor, whether simply verifying correct results or fixing incorrect ones, can dramatically improve ZephIR’s accuracy. Through this workflow, ZephIR achieves state-of-the-art accuracy with minimal manual labor, even on a freely behaving C. elegans, where large deformations present a particularly challenging tracking problem.