Neural population control via deep image synthesis

Bashivan, Pouya; Kar, Kohitij; DiCarlo, James J.

doi:10.1126/science.aav9436

Cited by 331 publications

(312 citation statements)

References 34 publications

(54 reference statements)

Supporting

Mentioning

244

Contrasting

Order By: Relevance

“…Recently, a family of computational models has emerged in the form of convolutional deep neural networks (DNNs) that allow to simulate this hierarchical information processing. When trained on object recognition, DNNs show interesting commonalities with the primate ventral stream, with a progression of representations that is surprisingly similar to what is seen in monkeys and humans for brief stimulus presentations (Cadieu et al, 2014; Yamins et al, 2014; Güçlü and van Gerven, 2015; Kalfas et al, 2017, 2018; Pospisil et al, 2018; Bashivan et al, 2019), as such capturing important aspects of object recognition and perceived shape similarity (Yamins et al, 2014; Kubilius et al, 2016; Kalfas et al, 2018). The architecture of these computational models is composed of series of convolutional layers that perform local filtering operations, followed by fully connected layers, which gradually transforms pixel level inputs into a high-level representational space where object categories are linearly separable.…”

Section: Introductionmentioning

confidence: 77%

Deep Neural Networks Point to Mid-level Complexity of Rodent Object Vision

Vinken

Beeck

2020

Preprint

View full text Add to dashboard Cite

1In the last two decades rodents have been on the rise as a dominant model for visual 2 neuroscience. This is particularly true for earlier levels of information processing, but 3 high-profile papers have suggested that also higher levels of processing such as invariant 4 object recognition occur in rodents. Here we provide a quantitative and comprehensive 5 assessment of this claim by comparing a wide range of rodent behavioral and neural data 6 with convolutional deep neural networks. These networks have been shown to capture 7 the richness of information processing in primates through a succession of convolutional 8 and fully connected layers. We find that rodent object vision can be captured using 9 low to mid-level convolutional layers only, without any convincing evidence for the 10 need of higher layers known to simulate complex object recognition in primates. Our 11 approach also reveals surprising insights on assumptions made before, for example, that 12 the best performing animals would be the ones using the most complex representations 13 -which we show to likely be incorrect. Our findings suggest a road ahead for further 14 studies aiming at quantifying and establishing the richness of representations underlying 15 information processing in animal models at large. 16 1 Introduction 17 Up to one decade ago, macaque monkeys were uncontested as the primary animal model for 18 vision research targeting information processing at the cortical level. Since then, studies on 19 rodents have become increasingly popular, in particular because developments in neurotech-20 nology such as cell-level imaging and optogenetics capitalized upon the existence of genetic 21 models. A major question emerging from this evolution is how far we can take rodents as a 22 model for the more complex aspects of visual information processing. 23To answer this question, a series of high-profile studies have documented the ability of rats 24 to perform seemingly complex object recognition tasks. In these tasks, rats show the ability 25 to recognize objects despite various transformations in viewing conditions such as position, 26 size, and viewpoint. In the first landmark study, Zoccolan and colleagues (2009) trained rats 27 2 to discriminate two computer-graphics rendered objects under various changes in size and 28 rotation, and showed that the animals could successfully generalize to novel combinations 29 of these transformations. Several studies have subsequently expanded on this work using 30 similar stimuli (Tafazoli et al., 2012; Alemi-Neissi et al., 2013; Rosselli et al., 2015), and 31 recently it was found that the success of rats in these tasks depends on the complexity of 32 their strategy (Djurdjevic et al., 2018). In a study using natural stimuli, rats could generalize 33 learned category rules to new, unseen category exemplars that differ from trained exemplars 34 in complex ways (Vinken et al., 2014). Neurophysiological recordings have revealed neural 35 responses in lateral visual areas that might underlie these b...

show abstract

Section: Introductionmentioning

confidence: 77%

Deep Neural Networks Point to Mid-level Complexity of Rodent Object Vision

Vinken

Beeck

2020

Preprint

View full text Add to dashboard Cite

show abstract

“…In order to establish a more causal relationship between visual areas and behavior, it will be important to combine behavioral performance with neural activity manipulation. Neural networks models and the inception loop methodology will enable the characterization of the specific features that drive neurons in these different visual areas (Bashivan et al, 2019;Ponce et al, 2019;Walker et al, 2019) .…”

mentioning

confidence: 99%

Object manifold geometry across the mouse cortical visual hierarchy

Froudarakis

Cohen

Diamantaki

et al. 2020

Preprint

View full text Add to dashboard Cite

Despite variations in appearance we robustly recognize objects. Neuronal populations responding to objects presented under varying conditions form object manifolds and hierarchically organized visual areas are thought to untangle pixel intensities into linearly decodable object representations. However, the associated changes in the geometry of object manifolds along the cortex remain unknown. Using home cage training we showed that mice are capable of invariant object recognition. We simultaneously recorded the responses of thousands of neurons to measure the information about object identity available across the visual cortex and found that lateral visual areas LM, LI and AL carry more linearly decodable object identity information compared to other visual areas. We applied the theory of linear separability of manifolds, and found that the increase in classification capacity is associated with a decrease in the dimension and radius of the object manifold, identifying features of the population code that enable invariant object coding.

show abstract

“…These results show that XDream can efficiently create images that trigger high activations in a target unit without making assumptions about the type of images a unit may prefer and without any knowledge of the target model architecture or connectivity, suggesting that XDream may well be applicable to biological neurons. Furthermore, XDream generalizes across layers in a ConvNet, while different layers roughly correspond to areas along the ventral visual stream [17,32,33], suggesting that XDream may also generalize to several ventral stream areas. Consistent with this observation, results from [13] indicated that XDream can find optimized stimuli for V1 as well as inferior temporal cortex (IT) neurons.…”

Section: Plos Computational Biologymentioning

confidence: 99%

XDream: Finding preferred stimuli for visual neurons using generative networks and gradient-free optimization

Xiao

Kreiman

2020

PLoS Comput Biol

View full text Add to dashboard Cite

A longstanding question in sensory neuroscience is what types of stimuli drive neurons to fire. The characterization of effective stimuli has traditionally been based on a combination of intuition, insights from previous studies, and luck. A new method termed XDream (EXtending DeepDream with real-time evolution for activation maximization) combined a generative neural network and a genetic algorithm in a closed loop to create strong stimuli for neurons in the macaque visual cortex. Here we extensively and systematically evaluate the performance of XDream. We use ConvNet units as in silico models of neurons, enabling experiments that would be prohibitive with biological neurons. We evaluated how the method compares to brute-force search, and how well the method generalizes to different neurons and processing stages. We also explored design and parameter choices. XDream can efficiently find preferred features for visual units without any prior knowledge about them. XDream extrapolates to different layers, architectures, and developmental regimes, performing better than brute-force search, and often better than exhaustive sampling of >1 million images. Furthermore, XDream is robust to choices of multiple image generators, optimization algorithms, and hyperparameters, suggesting that its performance is locally near-optimal. Lastly, we found no significant advantage to problem-specific parameter tuning. These results establish expectations and provide practical recommendations for using XDream to investigate neural coding in biological preparations. Overall, XDream is an efficient, general, and robust algorithm for uncovering neuronal tuning preferences using a vast and diverse stimulus space. XDream is implemented in Python, released under the MIT License, and works on Linux, Windows, and MacOS.

show abstract

Neural population control via deep image synthesis

Cited by 331 publications

References 34 publications

Deep Neural Networks Point to Mid-level Complexity of Rodent Object Vision

Deep Neural Networks Point to Mid-level Complexity of Rodent Object Vision

Object manifold geometry across the mouse cortical visual hierarchy

XDream: Finding preferred stimuli for visual neurons using generative networks and gradient-free optimization

Contact Info

Product

Resources

About