Language Reconstruction with Brain Predictive Coding from fMRI Data
Keywords: fMRI-to-text decoding, predictive coding theory
Abstract: Many recent studies have shown that the perception of speech can be decoded from brain signals and subsequently reconstructed as continuous language. However, there is a lack of neurological basis for how the semantic information embedded within brain signals can be used more effectively to guide language reconstruction. Predictive coding theory suggests the human brain naturally engages in continuously predicting future words that span multiple timescales. This implies that the decoding of brain signals could potentially be associated with a predictable future. To explore the predictive coding theory within the context of language reconstruction, this paper proposes PredFT (FMRI-to-Text decoding with Predictive coding). PredFT consists of a main decoding network and a side network. The side network obtains brain predictive coding representation from related brain regions of interest (ROIs) with a self-attention module. This representation is then fused into the main decoding network for continuous language decoding. Experiments are conducted on two popular naturalistic language comprehension fMRI datasets. Results show that PredFT achieves current state-of-the-art decoding performance on several evaluation metrics. Additional observations on the selection of ROIs, along with the length and distance parameters in predictive coding further guide the adoption of predictive coding theory for language reconstruction.
Primary Area: applications to neuroscience & cognitive science
Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
Submission Guidelines: I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.
Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.
No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.
Submission Number: 6263
Loading