Comparing visual representations across human fMRI and computational vision

Abstract

Feedforward visual object perception recruits a cortical network that is assumed to be hierarchical, progressing from basic visual features to complete object representations. However, the nature of the intermediate features related to this transformation remains poorly understood. Here, we explore how well different computer vision recognition models account for neural object encoding across the human cortical visual pathway as measured using fMRI. These neural data, collected during the viewing of 60 images of real-world objects, were analyzed with a searchlight procedure as in Kriegeskorte, Goebel, and Bandettini (2006): Within each searchlight sphere, the obtained patterns of neural activity for all 60 objects were compared to model responses for each computer recognition algorithm using representational dissimilarity analysis (Kriegeskorte et al., 2008). Although each of the computer vision methods significantly accounted for some of the neural data, among the different models, the scale invariant feature transform (Lowe, 2004), encoding local visual properties gathered from "interest points," was best able to accurately and consistently account for stimulus representations within the ventral pathway. More generally, when present, significance was observed in regions of the ventral-temporal cortex associated with intermediate-level object perception. Differences in model effectiveness and the neural location of significant matches may be attributable to the fact that each model implements a different featural basis for representing objects (e.g., more holistic or more parts-based). Overall, we conclude that well-known computer vision recognition systems may serve as viable proxies for theories of intermediate visual object representation.

The brain of the beholder: honouring individual representational idiosyncrasies

“…HMAX is a computational model inspired by the neuroscience literature, which aims at characterising the representations along the visual hierarchy. Efforts have been made to compare the representational geometry of the HMAX model to the representational geometry of activity patterns along the visual ventral stream (Kriegeskorte, 2009;Kriegeskorte, Mur, Ruff, et al, 2008;Leeds, Seibert, Pyles, & Tarr, 2013). The classical implementation of the HMAX model so far fails to satisfactorily explain the categorical divisions of higher level object representations.…”

A method for real-time visual stimulus selection in the study of cortical object perception

“…The past success of SIFT as a model for mid-level visual representation in the brain (Leeds et al, 2013) lends the model to study of visual properties of interest for diverse visual classes, from the cars and mammals examined in our current study to faces, tools, dwelling-places and beyond. The SIFT measure groups stimuli according to a distance matrix for object pairs (Leeds et al, 2013). In our present work, we defined a Euclidean space based on the distance matrix using Matlab’s implementation of metric multidimensional scaling (MDS, Seber (1984)).…”

“…Leeds et al (2013) found that a SIFT-based representation of visual objects was the best match among several machine vision models in accounting for the neural encoding of objects in mid-level visual areas along the ventral visual pathway. The past success of SIFT as a model for mid-level visual representation in the brain (Leeds et al, 2013) lends the model to study of visual properties of interest for diverse visual classes, from the cars and mammals examined in our current study to faces, tools, dwelling-places and beyond. The SIFT measure groups stimuli according to a distance matrix for object pairs (Leeds et al, 2013).…”

“…The use of relatively small — one cubic centimeter — cortical regions make it more likely that our methods will reveal information regarding local neural selectivities for complex visual properties. This assumption is based on analyses that were successfully pursued on similar spatial scales in Leeds et al (2013), using 123-voxel searchlights.…”

Exploration of complex visual feature spaces for object perception

“…Leeds et al (2013) found that a SIFT-based representation of visual objects was the best match among several machine vision models in accounting for the neural encoding of objects in mid-level visual areas along the ventral visual pathway. The SIFT measure groups stimuli according to a distance matrix for object pairs (Leeds et al, 2013). In our present work, we defined a Euclidean space based on the distance matrix using Matlab's implementation of metric multidimensional scaling (MDS, Seber, 1984).…”