Multiscale Pixel Spatiotemporal Information Flows

Felix Yuran Zhou, Roshan Ravishankar

Keywords: causal discovery, dynamical systems, granger causality, information flows

TL;DR: computing pixel-to-pixel information flows in video

Abstract: We develop a formal algorithmic framework to compute multiscale pixel spatiotemporal information flows which capture, in an unbiased manner, salient causal relationships between pixels across space and time. Real spatiotemporal dynamical systems such as cellular morphodynamics are inherently complex, nonlinear and evolve over time in response to feedbacks. This makes it highly challenging to directly model, simulate, or fit observed phenomena from first principle physics. Oftentimes neither the salient variables nor the key relationships are known a priori to include in a mathematical model. Even if a model was possible, we may be limited in our ability to sample the necessary information for exact system identification and verification. Alternatively, causal measures have been developed to identify potential causal relationships statistically from only observational timeseries. However such measures have largely only been studied for unstructured 1D timeseries where objects-of-interest have been pre-segmented and tracked over time. This restricts their application either to analyse general video dynamics, where individual objects are impossible to define or difficult to segment, or to understand potential causal relationships between subparts of objects. Here we propose a formal definition of a pixel spatiotemporal information flow as a spatiotemporal derivative of a pixel intensity timeseries to extract the dense pixel-to-pixel information transfer in 2D + time videos using any desired 1D causal measure, in a general and multiscale manner. Applying our framework, we discover salient pixel-to-pixel information highways in videos of diverse phenomena spanning traffic and crowd flow, collision physics, fish swarming, moving camouflaged animals, human action, embryo development, cell division and cell migration.