CORnet: Modeling the Neural Mechanisms of Core Object Recognition

Abstract

Deep artificial neural networks with spatially repeated processing (a.k.a., deep convolutional ANNs) have been established as the best class of candidate models of visual processing in primate ventral visual processing stream. Over the past five years, these ANNs have evolved from a simple feedforward eight-layer architecture in AlexNet to extremely deep and branching NAS-Net architectures, demonstrating increasingly better object categorization performance and increasingly better explanatory power of both neural and behavioral responses. However, from the neuroscientist's point of view, the relationship between such very deep architectures and the ventral visual pathway is incomplete in at least two ways. On the one hand, current stateof-the-art ANNs appear to be too complex (e.g., now over 100 levels) compared with the relatively shallow cortical hierarchy (4-8 levels), which makes it difficult to map their elements to those in the ventral visual stream and to understand what they are doing. On the other hand, current state-of-the-art ANNs appear to be not complex enough in that they lack recurrent connections and the resulting neural response dynamics that are commonplace in the ventral visual stream. Here we describe our ongoing efforts to resolve both of these issues by developing a "CORnet" family of deep neural network architectures. Rather than just seeking high object recognition performance (as the state-of-the-art ANNs above), we instead try to reduce the model family to its most important elements and then gradually build new ANNs with recurrent and skip connections while monitoring both performance and the match between each new CORnet model and a large body of primate brain and behavioral data. We report here that our current best ANN model derived from this approach (CORnet-S) is among the top models on Brain-Score, a composite benchmark for comparing models to the brain, but is simpler than other deep ANNs in terms of the number of convolutions performed along the longest path of information processing in the model. All CORnet models are available at github.com/dicarlolab/CORnet, and we plan to update this manuscript and the available models in this family as they are produced. object recognition | deep neural networks | feedforward | recurrence Correspondence: qbilius@mit.edu (J.K), mschrimpf@mit.edu (M.S.), dicarlo@mit.edu (J.J.D.) * Equal contribution We do not criticize the machine learning and computer vi-Kubilius, Schrimpf et al. | bioRχiv | September 4, 2018 | 1-9

Brain-Score: Which Artificial Neural Network for Object Recognition is most Brain-Like?

“…To further map out the space of possible architectures and a baseline of neural, behavioral, and performance scores, we included an in-house-developed family of models with up to moderate ImageNet performance, termed BaseNets: lightweight AlexNet-like architectures with six convolutional layers and a single fully-connected layer, captured at various stages of training. Various hyperparameters were varied between BaseNets, such as the number of filter maps, nonlinearities, pooling, learning rate scheduling, and so on, and formed a basis for the CORnet family of models (Kubilius et al, 2018b). We also tested CORnet-S, a new model that was developed with the goal of rivaling the best models on Brain-Score while being significantly shallower than competitors by leveraging bottleneck architecture and recurrence (Kubilius et al, 2018b).…”

“…Various hyperparameters were varied between BaseNets, such as the number of filter maps, nonlinearities, pooling, learning rate scheduling, and so on, and formed a basis for the CORnet family of models (Kubilius et al, 2018b). We also tested CORnet-S, a new model that was developed with the goal of rivaling the best models on Brain-Score while being significantly shallower than competitors by leveraging bottleneck architecture and recurrence (Kubilius et al, 2018b). CORnet-S is composed of four recurrent areas with two to three convolutions each and a fully-connected layer at the end.…”

Large-scale hyperparameter search for predicting human brain responses in the Algonauts challenge

“…Among the first five models that have high IT predictivity and are common for the two datasets is CORnet-S ( Figure 3). CORnets (7) have an architecture that approximates the number and size of visual areas in the macaque brain. CORnet-S is a recurrent ANN with skip connections.…”

Intrinsic dynamics enhance temporal stability of stimulus representation along rodent visual cortical hierarchies

“…In fact, deep convolutional neural networks have emerged as successful models of the ventral stream (Yamins et al 2014), and authors investigating the limitations of purely feedforward architectures within this family have proposed including temporal dynamics and adaptive mechanisms (Vinken et al 2019), or recurrent computations (Kar et al 2019;Kietzmann et al 2019;Tang et al 2018). Indeed, it has been suggested that convolutional networks that excel in object recognition need to be very deep simply to approximate operations that could be implemented more efficiently by recurrent architectures (Kar et al 2019;Kubilius et al 2018;Liao and Poggio 2016). These theoretical developments point to the potential importance of intrinsic dynamics for the cortical representation of visual stimuli.…”