XDream: Finding preferred stimuli for visual neurons using generative networks and gradient-free optimization

Abstract

A longstanding question in sensory neuroscience is what types of stimuli drive neurons to fire. The characterization of effective stimuli has traditionally been based on a combination of intuition, insights from previous studies, and luck. A new method termed XDream (EXtending DeepDream with real-time evolution for activation maximization) combined a generative neural network and a genetic algorithm in a closed loop to create strong stimuli for neurons in the macaque visual cortex. Here we extensively and systematically evaluate the performance of XDream. We use ConvNet units as in silico models of neurons, enabling experiments that would be prohibitive with biological neurons. We evaluated how the method compares to brute-force search, and how well the method generalizes to different neurons and processing stages. We also explored design and parameter choices. XDream can efficiently find preferred features for visual units without any prior knowledge about them. XDream extrapolates to different layers, architectures, and developmental regimes, performing better than brute-force search, and often better than exhaustive sampling of >1 million images. Furthermore, XDream is robust to choices of multiple image generators, optimization algorithms, and hyperparameters, suggesting that its performance is locally near-optimal. Lastly, we found no significant advantage to problem-specific parameter tuning. These results establish expectations and provide practical recommendations for using XDream to investigate neural coding in biological preparations. Overall, XDream is an efficient, general, and robust algorithm for uncovering neuronal tuning preferences using a vast and diverse stimulus space. XDream is implemented in Python, released under the MIT License, and works on Linux, Windows, and MacOS.

The geometry of cortical representations of touch in rodents

“…These results showed also that the classical concept of a receptive field, which is predictive of the responses only when the responses are linear, must be revisited [45, 46]. Interestingly, this approach has even been validated by generating synthetic visual stimuli that are able to maximally activate real neurons [47, 48]. Similarly, accurate models of the somatosensory system could be used to synthesize artificial somatosensory stimuli to drive neuronal activity or even behavior.…”

High-performance Evolutionary Algorithms for Online Neuron Control

“…To simulate neuronal tuning function 𝑓 that an optimizer might encounter in a biological setting, we used units from pre-trained CNNs as models of visual neurons [19,32]. For the in vivo Evolution experiments, we aimed for optimizers that performed well with neurons across visual areas (including V1, V4, IT) and across different levels of signal-to-noise and single-neuron isolation.…”

“…Image search was effectuated by classic genetic algorithms (GAs), acting in the space of parametrized 3D shapes or GANs. Though the use of GAs was successful in this domain [32], it has not been tested comprehensively against modern optimizers, which motivated us to determine if we could improve performance on this front.…”

“…This problem features a unique set of challenges, for example, search dimensionality is very high (e.g. 𝑑 = 4096 in [24,28,32]), and the function 𝑓 (.) must be evaluated with considerable noise in neuronal responses.…”

Generative adversarial networks unlock new methods for cognitive science

“…Finally, XDREAM [60,61] targets the activity in monkey IT-neurons directly and evolves preferred stimuli through a combination of a genetic algorithm and a GAN in an online feedback loop. Using a GAN avoids the need to hand pick stimuli and lowers the risk of missing critical features when sampling from fully defined but more constrained parametric spaces.…”