Dynamical modeling for real-time inference of nonlinear latent factors in multiscale neural activity
Keywords: Multimodal deep learning, Missing data, Neuroscience, Real-time decoding
Abstract: Continuous real-time decoding of target variables from time-series data is needed for many applications across various domains including neuroscience. Further, these variables can be encoded across multiple time-series modalities such as discrete spiking activity and continuous field potentials that can have different timescales (i.e., sampling rates) and different probabilistic distributions, or can even be missing at some time-steps. Existing nonlinear models of multimodal neural activity do not support real-time decoding and do not address the different timescales or missing samples across modalities. Here, we develop a learning framework that can nonlinearly aggregate information across multiple time-series modalities with such distinct characteristics, while also enabling real-time decoding. This framework consists of 1) a multiscale encoder that nonlinearly fuses information after learning within-modality dynamics to handle different timescales and missing samples, 2) a multiscale dynamical backbone that extracts multimodal temporal dynamics and enables real-time decoding, and 3) modality-specific decoders to account for different probabilistic distributions across modalities. We further introduce smoothness regularization objectives on the learned dynamics to better decode smooth target variables such as behavioral variables and employ a dropout technique to increase the robustness for missing samples. We show that our model can aggregate information across modalities to improve target variable decoding in simulations and in a real multiscale brain dataset. Further, our method outperforms prior linear and nonlinear multimodal models.
Primary Area: applications to neuroscience & cognitive science
Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
Submission Guidelines: I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.
Reciprocal Reviewing: I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.
Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.
No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.
Submission Number: 12699
Loading