Evidence that recurrent circuits are critical to the ventral stream’s execution of core object recognition behavior

Abstract

Non-recurrent deep convolutional neural networks (DCNNs) are currently the best models of core object recognition; a behavior supported by the densely recurrent primate ventral stream, culminating in the inferior temporal (IT) cortex. Are these recurrent circuits critical to ventral stream's execution of this behavior? We reasoned that, if recurrence is critical, then primates should outperform feedforward-only DCNNs for some images, and that these images should require additional processing time beyond the feedforward IT response. Here we first used behavioral methods to discover hundreds of these "challenge" images. Second, using large-scale IT electrophysiology in animals performing core recognition tasks, we observed that behaviorally-sufficient, linearly-decodable object identity solutions emerged ∼30ms (on average) later in IT for challenge images compared to DCNN and primate performance-matched "control" images. We observed these same late solutions even during passive viewing. Third, consistent with a failure of feedforward computations, the behaviorally-critical late-phase IT population response patterns evoked by the challenge images were poorly predicted by DCNN activations. Interestingly, deeper CNNs better predicted these late IT responses, suggesting a functional equivalence between recurrence and additional nonlinear transformations. Our results argue that automatically-evoked recurrent circuits are critical even for rapid object identification. By precisely comparing current DCNNs, primate behavior and IT population dynamics, we provide guidance for future recurrent model development.

Beyond the feedforward sweep: feedback computations in the visual cortex

“…At the neurophysiology level, it has been shown that the early response of neurons in intermediate and higher visual areas contains enough information for decoding image category almost readily from the onset of the visual response both during passive 12,13 and active 14 presentations. Consistent with this idea, a recent monkey electrophysiology study has also shown that images that are behaviorally more difficult to classify by human observers tend to take longer to be reliably decoded, possibly requiring additional feedback processes beyond this initial response 15 …”

“…Consistent with this idea, a recent monkey electrophysiology study has also shown that images that are behaviorally more difficult to classify by human observers tend to take longer to be reliably decoded, possibly requiring additional feedback processes beyond this initial response. 15 Human observers make recognition mistakes under these conditions, but these errors do not appear to be randomly distributed across images as would be expected from motor errors or guessing. Instead, there appears to be a systematic pattern of behavioral decisions-with some images being consistently classified correctly or incorrectly across human observers.…”

“…3C). 15 Recurrent computations are not only relevant for recognition, but they can also help solve other problems. We mentioned earlier the challenges in image segmentation in connectomics with purely feedforward architectures.…”

Intrinsic dynamics enhance temporal stability of stimulus representation along rodent visual cortical hierarchies

“…In fact, deep convolutional neural networks have emerged as successful models of the ventral stream (Yamins et al 2014), and authors investigating the limitations of purely feedforward architectures within this family have proposed including temporal dynamics and adaptive mechanisms (Vinken et al 2019), or recurrent computations (Kar et al 2019;Kietzmann et al 2019;Tang et al 2018). Indeed, it has been suggested that convolutional networks that excel in object recognition need to be very deep simply to approximate operations that could be implemented more efficiently by recurrent architectures (Kar et al 2019;Kubilius et al 2018;Liao and Poggio 2016). These theoretical developments point to the potential importance of intrinsic dynamics for the cortical representation of visual stimuli.…”

“…Recently it has been shown that models of the ventral stream based on deep convolutional neural networks can be improved in their predictive power for perception and neural activity by including adaptation mechanisms (Vinken et al 2019) and recurrent processing (Kar et al 2019;Kietzmann et al 2019;Tang et al 2018). Moreover, a progressive increase of the importance of intrinsic processing along the ventral stream may be expected, given that intrinsic temporal scales increase along various cortical hierarchies in primates and rodents (Chaudhuri et al 2015;Himberger et al 2018;Murray et al 2014;Runyan et al 2017).…”