Convolutional Neural Networks as a Model of the Visual System: Past, Present, and Future

Abstract

Convolutional neural networks (CNNs) were inspired by early findings in the study of biological vision. They have since become successful tools in computer vision and state-of-the-art models of both neural activity and behavior on visual tasks. This review highlights what, in the context of CNNs, it means to be a good model in computational neuroscience and the various ways models can provide insight. Specifically, it covers the origins of CNNs and the methods by which we validate them as models of biological vision. It then goes on to elaborate on what we can learn about biological vision by understanding and experimenting on CNNs and discusses emerging opportunities for the use of CNNS in vision research beyond basic object recognition. This is the author's final version. This article has been accepted for publication in the Journal ofCognitive Neuroscience.

Diverse deep neural networks all predict human IT well, after training and fitting

“…The advent of high-performing object-recognition DNNs in computer vision has provided visual neuroscience with unprecedentedly good models for predicting visual responses in the human and non-human primate brain (Kietzmann et al, 2018;Kriegeskorte, 2015;Lindsay, 2020;Schrimpf et al, 2018;Yamins et al, 2014). The deep learning modelling framework promises much more, beyond evaluating the effectiveness of off-the-shelf feedforward DNNs trained on computer vision tasks.…”

“…Recently, deep neural networks (DNN) using feedforward hierarchies of convolutional features to process images have reached and even surpassed human category-level recognition performance (He et al, 2016;Kietzmann et al, 2018;Lindsay, 2020;Russakovsky et al, 2015;Yamins & DiCarlo, 2016). Despite being developed as computer vision tools, DNNs trained to recognise objects in images are also unsurpassed at predicting how natural images are represented in high-level ventral visual areas of the human and non-human primate brain (Agrawal et al, 2014;Bashivan et al, 2019;Cadieu et al, 2014;Cichy et al, 2016;Devereux et al, 2018;Eickenberg et al, 2017;Güçlü & van Gerven, 2015;Horikawa & Kamitani, 2017;Kubilius et al, 2018;Lindsay, 2020;Ponce et al, 2019;Schrimpf et al, 2018;Xu & Vaziri-Pashkam, 2020;Yamins & DiCarlo, 2016). There is some variability in the accuracy with which different recent DNNs can predict high-level visual representations Xu & Vaziri-Pashkam, 2020;Zeman et al, 2020), despite broadly high performance.…”

Comparing memory capacity across stimuli requires maximally dissimilar foils: Using deep convolutional neural networks to understand visual working memory capacity for real-world objects

“…To address this, we make use of deep convolutional neural networks to achieve this goal. Recently deep convolutional neural networks have reached very high performance on object recognition tasks, reaching, and even surpassing in limited circumstances, human category-level recognition performance (He et al, 2016;Kietzmann et al, 2018;Lindsay, 2020;Russakovsky et al, 2015;Yamins & DiCarlo, 2016). Furthermore, evidence suggests that these networks capture some aspects of the neural representations of visual information, as they have been shown to account for neural data recorded in higher-level visual areas of the human and non-human primate brain (Kubilius et al 2018, Khaligh-Razavi & Kriegeskorte, 2014Yamins et al, 2014;Güçlü & van Gerven, 2015;Kar et al, 2019).…”

Attention in Psychology, Neuroscience, and Machine Learning

“…The architecture of convolutional neural networks is loosely based on the mammalian visual system (Lindsay, 2020). At each layer, a bank of filters is applied to the activity of the layer below (in the first layer this is the image).…”