Go to ICLR 2025 Conference homepageLinBridge: A Learnable Framework for Interpreting Nonlinear Neural Encoding Models
Neural encoding of artificial neural networks (ANNs) aligns the computational representations of ANNs with brain responses, providing profound insights into the neural basis underpinning information processing in the human brain. Current neural encoding studies primarily employ linear encoding models for interpretability, despite the prevalence of nonlinear neural responses. This leads to a growing interest in developing nonlinear encoding models that retain interpretability. To address this problem, we propose LinBridge, a learnable and flexible framework based on Jacobian analysis for interpreting nonlinear encoding models. LinBridge posits that the nonlinear mapping between ANN representations and neural responses can be factorized into a linear inherent component that approximates the complex nonlinear relationship, and a mapping bias that captures sample-selective nonlinearity. The Jacobian matrix, which reflects output change rates relative to input, enables the analysis of sample-selective mapping in nonlinear models. LinBridge employs a self-supervised learning strategy to extract both the linear inherent component and nonlinear mapping biases from the Jacobian matrices of the test set, allowing it to adapt effectively to various nonlinear encoding models. We validate the LinBridge framework in the scenario of neural visual encoding, using computational visual representations from CLIP-ViT to predict brain activity recorded via functional magnetic resonance imaging (fMRI). Our experimental results demonstrate that: 1) the linear inherent component extracted by LinBridge accurately reflects the complex mappings of nonlinear neural encoding models; 2) the sample-selective mapping bias elucidates the variability of nonlinearity across different levels of the visual processing hierarchy. This study not only introduces a novel tool for interpreting nonlinear neural encoding models but also provides novel evidence regarding the distribution of hierarchical nonlinearity within the visual cortex.