Disentangling Behaviorally Relevant Latent Dynamics in Multimodal Neural Time-series

ICLR 2026 Conference Submission22025 Authors

Keywords: Multimodal Deep Learning, Nonlinear Dynamical Modeling, Neural-Behavioral Modeling

Abstract: Multimodal neural data can enable a more complete understanding of brain dynamics underlying behavior. Modalities such as neuronal spiking activity, local field potentials (LFPs), and behavioral signals capture diverse spatiotemporal aspects of brain processes. By leveraging these complementary strengths, multimodal neural fusion can provide a unified, rich representation of brain-behavior processes and address the limitations of single-modality analyses, such as incomplete or noisy data. While recent works have jointly modeled behavior and neural data to disentangle sources of variability, they largely rely on latent variable models that use a single modality of neural data. Here we develop a nonlinear dynamical model, termed BREM-NET, that integrates behavioral signals and multiple neural modalities—such as LFPs and spike counts—with distinct statistical characteristics and temporal resolutions into a unified framework, and performs multimodal neural fusion during inference. In two independent public multimodal neural datasets, we show that BREM-NET nonlinearly fuses information across neural modalities while also disentangling behaviorally relevant multimodal neural dynamics. Doing so results in inferring more accurate disentangled latent dynamics, as reflected in enhanced behavior decoding and neural prediction compared to multimodal baselines. Furthermore, BREM-NET enables disentanglement and multimodal fusion even when different neural time-series modalities are asynchronous and have distinct temporal resolutions, which is a major challenge in real-world neural recordings. This framework provides a new tool for studying behaviorally relevant neural computations across different spatiotemporal scales of brain activity.

Primary Area: applications to neuroscience & cognitive science

Submission Number: 22025