Curvature as an Organizing Principle of Mid-level Visual Representation: A Semantic-preference Mapping Approach

Shi Pui Donald Li, Michael Bonner

Keywords: Mid-level representation, Curvature, Encoding model, fMRI

Abstract: A central challenge in visual neuroscience is understanding the mid-level representations of the ventral stream. We used a novel, data-driven approach (semantic-preference mapping) combined with an image-statistics approach (curvature index) to characterize the mid-level features of category-selective visual regions. First, we fit voxelwise encoding models using a deep convolutional neural network (DCNN) to predict scene-evoked fMRI responses in object-selective and scene-selective regions. We then performed semantic-preference mapping to examine how the responses of these encoding models changed when specific object classes were removed from natural images. This analysis motivated the hypothesis that object-selective cortex model is best predicted by mid-level features with curved contours, while scene-selective cortex model is best predicted by mid-level features with rectilinear contours. We further developed an image-computable model that outputs a summary statistic for the prevalence of curved contours in local image patches, and we used this model to demonstrate the importance of curvature-preferences for linking DCNN representations with the representations of category-selective cortex models. Overall, our findings suggest that curvature is a key property of the mid-level representations that are shared between DCNNs and category-selective cortex models of the ventral visual stream.