Identifying natural images from human brain activity

Kay, Kendrick; Naselaris, Thomas; Prenger, Ryan; Gallant, Jack L.

doi:10.1038/nature06713

Cited by 1,164 publications

(1,416 citation statements)

References 30 publications

Supporting

Mentioning

1,379

Contrasting

Unclassified

Order By: Relevance

“…Our approach is analogous in some ways to research that focuses on lower-level visual features of picture stimuli to analyze fMRI activation associated with viewing the picture (O'Toole et al, 2005;Hardoon et al, 2007;Kay et al, 2008). A similar generative classifier is used by Kay et al (2008) where they estimate a receptive-field model for each voxel and classify an activation pattern in terms of its similarity to the predicted brain activity.…”

Section: Classifier Modelmentioning

confidence: 99%

“…A similar generative classifier is used by Kay et al (2008) where they estimate a receptive-field model for each voxel and classify an activation pattern in terms of its similarity to the predicted brain activity. Our work differs from these efforts, in that we focus on encodings of more abstract semantic features signified by words and predict brain activity based on these semantic features, rather than on visual features that encode visual properties.…”

Section: Classifier Modelmentioning

confidence: 99%

See 1 more Smart Citation

Quantitative modeling of the neural representation of objects: How semantic feature norms can account for fMRI activation

2011

View full text Add to dashboard Cite

Recent multivariate analyses of fMRI activation have shown that discriminative classifiers such as Support Vector Machines (SVM) are capable of decoding fMRI-sensed neural states associated with the visual presentation of categories of various objects. However, the lack of a generative model of neural activity limits the generality of these discriminative classifiers for understanding the underlying neural representation. In this study, we propose a generative classifier that models the hidden factors that underpin the neural representation of objects, using a multivariate multiple linear regression model. The results indicate that object features derived from an independent behavioral feature norming study can explain a significant portion of the systematic variance in the neural activity observed in an object-contemplation task. Furthermore, the resulting regression model is useful for classifying a previously unseen neural activation vector, indicating that the distributed pattern of neural activities encodes sufficient signal to discriminate differences among stimuli. More importantly, there appears to be a double dissociation between the two classifier approaches and within-versus between-participants generalization. Whereas an SVM-based discriminative classifier achieves the best classification accuracy in within-participants analysis, the generative classifier outperforms an SVM-based model which does not utilize such intermediate representations in between-participants analysis. This pattern of results suggests the SVM-based classifier may be picking up some idiosyncratic patterns that do not generalize well across participants and that good generalization across participants may require broad, large-scale patterns that are used in our set of intermediate semantic features. Finally, this intermediate representation allows us to extrapolate the model of the neural activity to previously unseen words, which cannot be done with a discriminative classifier.

show abstract

Section: Classifier Modelmentioning

confidence: 99%

Section: Classifier Modelmentioning

confidence: 99%

Quantitative modeling of the neural representation of objects: How semantic feature norms can account for fMRI activation

2011

View full text Add to dashboard Cite

show abstract

“…This analysis rests on the assumption that the distribution of feature-selective neurons-in this case the distribution of orientation-selective columns-is not uniform across a given visual area (Boynton 2005a;Kamitani and Tong 2005;Swisher et al 2010). Due to this nonuniform distribution of neural selectivity, a given voxel may contain more neurons tuned to one particular orientation, giving rise to a Gaussian-shaped response profile across orientations, which we refer to as the VTF (Serences et al 2009; see also Kamitani and Tong 2005;Kay et al 2008;Miyawaki et al 2008).…”

Section: Analysis Of Feature-selective Vtfs In Each Roimentioning

confidence: 99%

“…A quantitative model was then used to estimate the value of each alternative on a trial-by-trial basis, and fMRI-based voxel tuning functions (or VTFs; Kay et al 2008;Serences et al 2009) were used to estimate the influence of value on the shape of population response profiles in early areas of visual cortex. The data suggest that response profiles in early visual cortex are selectively biased in favor of highvalue stimulus features, particularly in V1, which contains a high proportion of orientation-tuned cells.…”

Section: Introductionmentioning

confidence: 99%

Population Response Profiles in Early Visual Cortex Are Biased in Favor of More Valuable Stimuli

Serences

Saproo

2010

Journal of Neurophysiology

104

View full text Add to dashboard Cite

Serences JT, Saproo S. Population response profiles in early visual cortex are biased in favor of more valuable stimuli. J Neurophysiol 104: 76 -87, 2010. First published April 21, 2010 doi:10.1152/jn.01090.2009. Voluntary and stimulus-driven shifts of attention can modulate the representation of behaviorally relevant stimuli in early areas of visual cortex. In turn, attended items are processed faster and more accurately, facilitating the selection of appropriate behavioral responses. Information processing is also strongly influenced by past experience and recent studies indicate that the learned value of a stimulus can influence relatively late stages of decision making such as the process of selecting a motor response. However, the learned value of a stimulus can also influence the magnitude of cortical responses in early sensory areas such as V1 and S1. These early effects of stimulus value are presumed to improve the quality of sensory representations; however, the nature of these modulations is not clear. They could reflect nonspecific changes in response amplitude associated with changes in general arousal or they could reflect a bias in population responses so that high-value features are represented more robustly. To examine this issue, subjects performed a two-alternative forced choice paradigm with a variableinterval payoff schedule to dynamically manipulate the relative value of two stimuli defined by their orientation (one was rotated clockwise from vertical, the other counterclockwise). Activation levels in visual cortex were monitored using functional MRI and feature-selective voxel tuning functions while subjects performed the behavioral task. The results suggest that value not only modulates the relative amplitude of responses in early areas of human visual cortex, but also sharpens the response profile across the populations of feature-selective neurons that encode the critical stimulus feature (orientation). Moreover, changes in space-or feature-based attention cannot easily explain the results because representations of both the selected and the unselected stimuli underwent a similar feature-selective modulation. This sharpening in the population response profile could theoretically improve the probability of correctly discriminating high-value stimuli from low-value alternatives.

show abstract

“…Moreover, researchers can already interpret a person's neural activity from functional magnetic resonance imaging scans at a rudimentary level 1 -that the individual is thinking of a person, say, rather than a car.…”

mentioning

confidence: 99%

Four ethical priorities for neurotechnologies and AI

Yuste

Goering

Arcas

et al. 2017

Nature

322

217

View full text Add to dashboard Cite

A man with a spinal-cord injury (right) prepares for a virtual cycle race in which competitors steer avatars using brain signals. COMMENT © 2 0 1 7 M a c m i l l a n P u b l i s h e r s L i m i t e d , p a r t o f S p r i n g e r N a t u r e . A l l r i g h t s r e s e r v e d .example. Moreover, researchers can already interpret a person's neural activity from functional magnetic resonance imaging scans at a rudimentary level 1 -that the individual is thinking of a person, say, rather than a car.It might take years or even decades until BCI and other neurotechnologies are part of our daily lives. But technological developments mean that we are on a path to a world in which it will be possible to decode people's mental processes and directly manipulate the brain mechanisms underlying their intentions, emotions and decisions; where individuals could communicate with others simply by thinking; and where powerful computational systems linked directly to people's brains aid their interactions with the world such that their mental and physical abilities are greatly enhanced.Such advances could revolutionize the treatment of many conditions, from brain injury and paralysis to epilepsy and schizophrenia, and transform human experience for the better. But the technology could also exacerbate social inequalities and offer corporations, hackers, governments or anyone else new ways to exploit and manipulate people. And it could profoundly alter some core human characteristics: private mental life, individual agency and an understanding of individuals as entities bound by their bodies.It is crucial to consider the possible ramifications now.The Morningside Group comprises neuroscientists, neurotechnologists, clinicians, ethicists and machine-intelligence engineers. It includes representatives from Google and Kernel (a neurotechnology start-up in Los Angeles, California); from international brain projects; and from academic and research institutions in the United States, Canada, Europe, Israel, China, Japan and Australia. We gathered at a workshop sponsored by the US National Science Foundation at Columbia University, New York, in May 2017 to discuss the ethics of neurotechnologies and machine intelligence.We believe that existing ethics guidelines are insufficient for this realm 2 . These include the Declaration of Helsinki, a statement of ethical principles first established in 1964 for medical research involving human subjects (go.nature.com/2z262ag); the Belmont Report, a 1979 statement crafted by the US National Commission for the Protection of Human Subjects of Biomedical and Behavioural Research (go.nature.com/2hrezmb); and the Asilomar artificial intelligence (AI) statement of cautionary principles, published early this year and signed by business leaders and AI researchers, among others (go.nature.com/2ihnqac).To begin to address this deficit, here we lay out recommendations relating to four areas of concern: privacy and consent; agency and identity; augmentation; and bias. Different nations and people of varying re...

show abstract

Identifying natural images from human brain activity

Cited by 1,164 publications

References 30 publications

Quantitative modeling of the neural representation of objects: How semantic feature norms can account for fMRI activation

Quantitative modeling of the neural representation of objects: How semantic feature norms can account for fMRI activation

Population Response Profiles in Early Visual Cortex Are Biased in Favor of More Valuable Stimuli

Four ethical priorities for neurotechnologies and AI

Contact Info

Product

Resources

About