Metamers of the ventral stream

Freeman, Jeremy; Simoncelli, Eero P.

doi:10.1038/nn.2889

Cited by 589 publications

(871 citation statements)

References 49 publications

Supporting

Mentioning

825

Contrasting

Order By: Relevance

“…To control for low-level confounds, we tested the (trivial) pixel model, as well as SIFT, a simple baseline computer vision model (30). We also evaluated a V1-like Gabor-based model (25), a V2-like conjunction-ofGabors model (31), and HMAX (17,28), a model targeted at explaining higher ventral cortex and that has receptive field sizes similar to those observed in IT. The HMAX model can be trained in a domain-specific fashion, and to give it the best chance of success, we performed this training using the benchmark images themselves (see SI Text for more information on the comparison models).…”

Section: Resultsmentioning

confidence: 99%

Performance-optimized hierarchical models predict neural responses in higher visual cortex

Yamins

Hong

Cadieu

et al. 2014

Proc. Natl. Acad. Sci. U.S.A.

1,618

1,797

View full text Add to dashboard Cite

The ventral visual stream underlies key human visual object recognition abilities. However, neural encoding in the higher areas of the ventral stream remains poorly understood. Here, we describe a modeling approach that yields a quantitatively accurate model of inferior temporal (IT) cortex, the highest ventral cortical area. Using high-throughput computational techniques, we discovered that, within a class of biologically plausible hierarchical neural network models, there is a strong correlation between a model's categorization performance and its ability to predict individual IT neural unit response data. To pursue this idea, we then identified a high-performing neural network that matches human performance on a range of recognition tasks. Critically, even though we did not constrain this model to match neural data, its top output layer turns out to be highly predictive of IT spiking responses to complex naturalistic images at both the single site and population levels. Moreover, the model's intermediate layers are highly predictive of neural responses in the V4 cortex, a midlevel visual area that provides the dominant cortical input to IT. These results show that performance optimization-applied in a biologically appropriate model classcan be used to build quantitative predictive models of neural processing.computational neuroscience | computer vision | array electrophysiology R etinal images of real-world objects vary drastically due to changes in object pose, size, position, lighting, nonrigid deformation, occlusion, and many other sources of noise and variation. Humans effortlessly recognize objects rapidly and accurately despite this enormous variation, an impressive computational feat (1). This ability is supported by a set of interconnected brain areas collectively called the ventral visual stream (2, 3), with homologous areas in nonhuman primates (4). The ventral stream is thought to function as a series of hierarchical processing stages (5-7) that encode image content (e.g., object identity and category) increasingly explicitly in successive cortical areas (1,8,9). For example, neurons in the lowest area, V1, are well described by Gabor-like edge detectors that extract rough object outlines (10), although the V1 population does not show robust tolerance to complex image transformations (9). Conversely, rapidly evoked population activity in top-level inferior temporal (IT) cortex can directly support realtime, invariant object categorization over a wide range of tasks (11,12). Midlevel ventral areas-such as V4, the dominant cortical input to IT-exhibit intermediate levels of object selectivity and variation tolerance (12-14).Significant progress has been made in understanding lower ventral areas such as V1, where conceptually compelling models have been discovered (10). These models are also quantitatively accurate and can predict response magnitudes of individual neuronal units to novel image stimuli. Higher ventral cortical areas, especially V4 and IT, have been much more difficult to understand. Al...

show abstract

Section: Resultsmentioning

confidence: 99%

Performance-optimized hierarchical models predict neural responses in higher visual cortex

Yamins

Hong

Cadieu

et al. 2014

Proc. Natl. Acad. Sci. U.S.A.

1,618

1,797

View full text Add to dashboard Cite

show abstract

“…Considering that curvature can also be regarded as a higher-order combinatory feature of Gabor filter output, it may be that there is shared tuning between these contour fragments and textures. However, texture synthesis does not capture global image structures and cannot reproduce inhomogeneous image features such as object contours (20), although it may explain visual discriminability of those features in peripheral visual field (21,41,42). It is therefore also plausible that tuning to geometric shapes or curvatures is hidden in the unexplained variances in the neural responses recorded in this study, or that they can be attributed to a different population of V4 neurons tuned to those features.…”

Section: Discussionmentioning

confidence: 97%

“…Their algorithm is particularly inspiring because PS statistics use filters and computations that share biological properties. It was recently shown, for example, that a version of their synthesis algorithm can generate perceptually indistinguishable visual images (visual metamers) (21) and that naturalistic textures incorporating these summary statistics strongly activate neurons in V2, compared with noise images lacking these features (7).…”

mentioning

confidence: 99%

Image statistics underlying natural texture selectivity of neurons in macaque V4

Okazawa

Tajima

Komatsu

2014

Proc. Natl. Acad. Sci. U.S.A.

144

171

View full text Add to dashboard Cite

Our daily visual experiences are inevitably linked to recognizing the rich variety of textures. However, how the brain encodes and differentiates a plethora of natural textures remains poorly understood. Here, we show that many neurons in macaque V4 selectively encode sparse combinations of higher-order image statistics to represent natural textures. We systematically explored neural selectivity in a high-dimensional texture space by combining texture synthesis and efficient-sampling techniques. This yielded parameterized models for individual texture-selective neurons. The models provided parsimonious but powerful predictors for each neuron's preferred textures using a sparse combination of image statistics. As a whole population, the neuronal tuning was distributed in a way suitable for categorizing textures and quantitatively predicts human ability to discriminate textures. Together, we suggest that the collective representation of visual image statistics in V4 plays a key role in organizing the natural texture perception.texture perception | material perception | visual area V4 | single-cell recording | image analysis I n the visual world, objects are characterized in part by their shapes, but also by their textures (1). The wide variety of textures we experience enables us to segment objects from backgrounds, perceive object properties, and recognize materials. It is well established that the representation of complex shapes and contours is gradually built up along the ventral visual pathway (2-5). On the other hand, how textural information is processed in the cortex is largely unknown, although some recent studies have examined the representation of natural textures and surface properties in the macaque ventral visual areas (6-11). Because, unlike contours, textures cannot be described based on combinations of edge fragments, we need to consider different underlying cortical processing.In contrast to the limited knowledge available from physiology, computational descriptions of textures have been extensively developed in the fields of psychophysics (12-16) and computer vision (17)(18)(19). In one such description, Portilla and Simoncelli (20) proposed that textures could be represented using an ensemble of summary statistics, including features derived from the luminance histogram and the amplitudes of the outputs of Gabor-like filters, as well as higher-order statistics such as the correlations across the filter outputs (see Fig. 3 for details; hereafter, we call this collection of statistics "PS statistics," using the authors' initials). Portilla and Simoncelli successfully generated new textures that were visually indistinguishable from the originals solely by making their PS statistics identical. Their algorithm is particularly inspiring because PS statistics use filters and computations that share biological properties. It was recently shown, for example, that a version of their synthesis algorithm can generate perceptually indistinguishable visual images (visual metamers) (21) and that naturalisti...

show abstract

“…Crowding occurs when the perception of a target stimulus, which would be visible if presented in isolation, is impaired by other surrounding items in close spatial proximity (Bouma, 1970;Pelli & Tillman, 2008). Although there are many models of crowding (for a review, see Whitney & Levi, 2011), a popular account characterises it as a pooling process that regularises the noisy representation of position in the periphery by averaging the target and flanker identities (Freeman & Simoncelli, 2011;Greenwood, Bex, & Dakin, 2009;Parkes, Lund, Angelucci, Solomon, & Morgan, 2001). Functionally, then, crowding may reflect the blurring and loss of information across space in the way that OSM reflects the blurring and loss of information across time.…”

mentioning

confidence: 99%

Categorical information influences conscious perception: An interaction between object-substitution masking and repetition blindness

Goodhew

Greenwood

Edwards

2016

Atten Percept Psychophys

View full text Add to dashboard Cite

The visual system is constantly bombarded with dynamic input. In this context, the creation of enduring object representations presents a particular challenge. We used object-substitution masking (OSM) as a tool to probe these processes. In particular, we examined the effect of target-like stimulus repetitions on OSM. In visual crowding, the presentation of a physically identical stimulus to the target reduces crowding and improves target perception, whereas in spatial repetition blindness, the presentation of a stimulus that belongs to the same category (type) as the target impairs perception. Across two experiments, we found an interaction between spatial repetition blindness and OSM, such that repeating a same-type stimulus as the target increased masking magnitude relative to presentation of a different-type stimulus. These results are discussed in the context of the formation of object files. Moreover, the fact that the inducer only had to belong to the same Btype^as the target in order to exacerbate masking, without necessarily being physically identical to the target, has important implications for our understanding of OSM per se. That is, our results show the target is processed to a categorical level in OSM despite effective masking and, strikingly, demonstrate that this category-level content directly influences whether or not the target is perceived, not just performance on another task (as in priming).

show abstract

Metamers of the ventral stream

Cited by 589 publications

References 49 publications

Performance-optimized hierarchical models predict neural responses in higher visual cortex

Performance-optimized hierarchical models predict neural responses in higher visual cortex

Image statistics underlying natural texture selectivity of neurons in macaque V4

Categorical information influences conscious perception: An interaction between object-substitution masking and repetition blindness

Contact Info

Product

Resources

About