Revealing the multidimensional mental representations of natural objects underlying human similarity judgements

Hebart, Martin N.; Zheng, Charles; Pereira, Francisco; Baker, Chris I.

doi:10.1038/s41562-020-00951-3

Cited by 148 publications

(247 citation statements)

References 53 publications

(35 reference statements)

Supporting

Mentioning

237

Contrasting

Order By: Relevance

“…Here, we presented a first foray into this domain by showing that ecoset training leads to an increase in face-selective units in final network layers. Moving further into the domain of behavior, it will be of interest to perform in-depth tests of ecoset-trained networks (supervised or unsupervised) to compare their task performance and error distributions against human behavioral data ( 39 – 42 ).…”

Section: Resultsmentioning

confidence: 99%

An ecologically motivated image dataset for deep learning yields better models of human vision

Mehrer

Spoerer

Jones

et al. 2021

Proc. Natl. Acad. Sci. U.S.A.

102

105

View full text Add to dashboard Cite

Deep neural networks provide the current best models of visual information processing in the primate brain. Drawing on work from computer vision, the most commonly used networks are pretrained on data from the ImageNet Large Scale Visual Recognition Challenge. This dataset comprises images from 1,000 categories, selected to provide a challenging testbed for automated visual object recognition systems. Moving beyond this common practice, we here introduce ecoset, a collection of >1.5 million images from 565 basic-level categories selected to better capture the distribution of objects relevant to humans. Ecoset categories were chosen to be both frequent in linguistic usage and concrete, thereby mirroring important physical objects in the world. We test the effects of training on this ecologically more valid dataset using multiple instances of two neural network architectures: AlexNet and vNet, a novel architecture designed to mimic the progressive increase in receptive field sizes along the human ventral stream. We show that training on ecoset leads to significant improvements in predicting representations in human higher-level visual cortex and perceptual judgments, surpassing the previous state of the art. Significant and highly consistent benefits are demonstrated for both architectures on two separate functional magnetic resonance imaging (fMRI) datasets and behavioral data, jointly covering responses to 1,292 visual stimuli from a wide variety of object categories. These results suggest that computational visual neuroscience may take better advantage of the deep learning framework by using image sets that reflect the human perceptual and cognitive experience. Ecoset and trained network models are openly available to the research community.

show abstract

Section: Resultsmentioning

confidence: 99%

An ecologically motivated image dataset for deep learning yields better models of human vision

Mehrer

Spoerer

Jones

et al. 2021

Proc. Natl. Acad. Sci. U.S.A.

102

105

View full text Add to dashboard Cite

show abstract

“…Furthermore, unlike human adults, these models have no knowledge of drawing conventions (i.e. how one might typically draw a bird or a fish) and do not incorporate abstract semantic features into their similarity judgements (Hebart, Zheng, Pereira, & Baker, 2020). This new tool makes it possible to assess developmental changes in the visual features of children's drawings.…”

Section: Introductionmentioning

confidence: 99%

Parallel developmental changes in children's production and recognition of line drawings of visual concepts

Long¹,

Fan²,

R³

et al. 2021

Preprint

View full text Add to dashboard Cite

To what extent do visual concepts of dogs, cars, and clocks change across childhood? We hypothesized that as children progressively learn which features best distinguish visual concepts from one another, they also improve their ability to connect this knowledge with external representations. To examine this possibility, we investigated developmental changes in children's ability to produce and recognize drawings of common object categories. First, we recruited children aged 2-10 years to produce drawings of 48 categories via a free-standing kiosk in a children's museum, and we measured how recognizable these >37K drawings were using a deep convolutional neural network model of object recognition. Second, we recruited other children across the same age range to identify the drawn category in a subset of these drawings via "guessing games" at the same kiosk.We found consistent developmental gains in both children's ability to include diagnostic visual features in their drawings and in children's ability to use these features when recognizing other children's drawings. Our results suggest that children's ability to connect internal and external representations of visual concepts improves gradually across childhood and imply that developmental trajectories of visual concept learning may be more protracted than previously thought.

show abstract

“…Note that Bankson et al (2018) exploited two different datasets which we label with “(1)” and “(2)” in Figure 3. The number of images per dataset are as follows: (Cichy, Pantazis, & Oliva, 2014; Kriegeskorte, Mur, Ruff, et al, 2008; Mur et al, 2013): 92; (Bankson et al, 2018) 84 each; (Cichy, Khosla, Pantazis, Torralba, & Oliva, 2016; Cichy et al, 2019): 118; (Mohsenzadeh et al, 2019): 156; (Hebart et al, 2019, 2020): 1854. For each of these datasets except for Mohsenzadeh et al (2019), we additionally computed RDMs for group averages obtained from behavioral experiments.…”

Section: Applications and Resultsmentioning

confidence: 99%

“…For Mohsenzadeh et al (2019), no behavioral experiments had been conducted. For both datasets in Bankson et al (2018), and for Hebart et al (2020), no fMRI recordings were available. For display purposes, Hebart et al (2020) was downsampled to 200 conditions.…”

Section: Applications and Resultsmentioning

confidence: 99%

THINGSvision: a Python toolbox for streamlining the extraction of activations from deep neural networks

Muttenthaler

Hebart

2021

Preprint

Self Cite

View full text Add to dashboard Cite

Over the past decade, deep neural network (DNN) models have received a lot of attention due to their oftentimes near-human object classification performance and their excellent prediction of signals recorded from biological visual systems. To better understand the function of these networks and relate them to hypotheses about brain activity and behavior, researchers need to extract the activations to images across different DNN layers. The abundance of different DNN variants, however, can often be unwieldy, and the task of extracting DNN activations from different layers may be non-trivial and error-prone for someone without a strong computational background. Thus, researchers in the fields of cognitive science and computational neuroscience would benefit from a library or package that supports a user in the extraction task. THINGSvision is a new Python module that aims at closing this gap by providing a simple and unified tool for extracting layer activations for a wide range of pretrained and randomly-initialized neural network architectures, even for users with little to no programming experience. We demonstrate the general utility of THINGSvision on a number of functional MRI and behavioral datasets, using representational similarity analysis which can be performed as an integral part of THINGSvision to relate DNN activations to empirical data. Together, THINGSvision enables researchers across diverse fields to extract features in a streamlined manner for their custom image dataset, thereby improving the ease of relating DNNs, brain activity, and behavior, and improving the reproducibility of findings in these research fields.

show abstract

Revealing the multidimensional mental representations of natural objects underlying human similarity judgements

Cited by 148 publications

References 53 publications

An ecologically motivated image dataset for deep learning yields better models of human vision

An ecologically motivated image dataset for deep learning yields better models of human vision

Parallel developmental changes in children's production and recognition of line drawings of visual concepts

THINGSvision: a Python toolbox for streamlining the extraction of activations from deep neural networks

Contact Info

Product

Resources

About