Visual prototypes in the ventral stream are attuned to complexity and gaze behavior

Abstract

Early theories of efficient coding suggested the visual system could compress the world by learning to represent features where information was concentrated, such as contours. This view was validated by the discovery that neurons in posterior visual cortex respond to edges and curvature. Still, it remains unclear what other information-rich features are encoded by neurons in more anterior cortical regions (e.g., inferotemporal cortex). Here, we use a generative deep neural network to synthesize images guided by neuronal responses from across the visuocortical hierarchy, using floating microelectrode arrays in areas V1, V4 and inferotemporal cortex of two macaque monkeys. We hypothesize these images (“prototypes”) represent such predicted information-rich features. Prototypes vary across areas, show moderate complexity, and resemble salient visual attributes and semantic content of natural images, as indicated by the animals’ gaze behavior. This suggests the code for object recognition represents compressed features of behavioral relevance, an underexplored aspect of efficient coding.

High-performance Evolutionary Algorithms for Online Neuron Control

“…For each network, we picked 5 layers of different depths. It has been noted that units from shallow-to-deep layers prefer features of increasing complexity [22], similar to that in the ventral stream cortical hierarchy [28]. For detailed information about these networks and their layers, see Sec.…”

“…We reasoned that each optimization trajectory should lead to a local optimum of the tuning function of each given CNNs unit or cortical neuron, encoding relevant visual features. Though different neurons or units prefer diverse visual attributes of different complexity [28], it was possible that the preferred visual features populated certain subspaces in the latent space of our generator. We first collected the in silico Evolution trajectories of 𝑁 = 1050 runs, across all conditions, and represented each trajectory by the mean code from the final generation, i.e.…”

“…As for in vivo evolutions, we examined the collection of evolution trajectories (𝑁 = 264 from 2 monkeys, [28]) and projected them onto the Hessian eigenframe as above. We found that they preferentially aligned with the top eigenspace than the lower eigenspace (Fig.…”

“…This optimization approach has been applied to neurons in visual areas V4 and inferotemporal cortex (IT) [4,16,24,28,30,33]. Image search was effectuated by classic genetic algorithms (GAs), acting in the space of parametrized 3D shapes or GANs.…”

“…This problem features a unique set of challenges, for example, search dimensionality is very high (e.g. 𝑑 = 4096 in [24,28,32]), and the function 𝑓 (.) must be evaluated with considerable noise in neuronal responses.…”