Recurrent neural networks can explain flexible trading of speed and accuracy in biological vision

Abstract

Deep feedforward neural network models of vision dominate in both computational neuroscience and engineering. The primate visual system, by contrast, contains abundant recurrent connections. Recurrent signal flow enables recycling of limited computational resources over time, and so might boost the performance of a physically finite brain or model. Here we show: (1) Recurrent convolutional neural network models outperform feedforward convolutional models matched in their number of parameters in large-scale visual recognition tasks on natural images. (2) Setting a confidence threshold, at which recurrent computations terminate and a decision is made, enables flexible trading of speed for accuracy. At a given confidence threshold, the model expends more time and energy on images that are harder to recognise, without requiring additional parameters for deeper computations. (3) The recurrent model’s reaction time for an image predicts the human reaction time for the same image better than several parameter-matched and state-of-the-art feedforward models. (4) Across confidence thresholds, the recurrent model emulates the behaviour of feedforward control models in that it achieves the same accuracy at approximately the same computational cost (mean number of floating-point operations). However, the recurrent model can be run longer (higher confidence threshold) and then outperforms parameter-matched feedforward comparison models. These results suggest that recurrent connectivity, a hallmark of biological visual systems, may be essential for understanding the accuracy, flexibility, and dynamics of human visual recognition.

Simulating the emergence of the Visual Word Form Area: Recycling a convolutional neural network for reading

“…It is also consistent with recent empirical results indicating that the main change induced by literacy is a sensitivity to letter shapes and their precise locations [40,41]. As also concluded by others [35], the simulation of recurrent interactions may be needed to capture the full temporal profile of word-related activations and to mimic fMRI signals.…”

“…A failure to capture some of these properties with a simple feedforward convolutional neural network would be interesting inasmuch as it may point to the need for additional properties, for instance recurrent and/or top-down connections [12,14,31,35].…”

“…These limitations of CNNs as models of the visual cortex are severe, and yet astonishingly, it is now well documented that when trained on a large number of categories, not only can CNNs reach human-level performance on visual classification tasks, but their single-unit responses can also predict neural, fMRI and MEG activation patterns in the visual cortex of human and non-human primates [35,[75][76][77][78][79][80]. Our choice of CNN architectures for modelling vOTC is thus conservative, and since the present models already mimic several known properties of the reading system (e.g.…”

Individual differences among deep neural network models

“…The observation of increased differences with increasing network depth is in line with findings from the domain of machine learning that compared network representations using methods related to CCA (svCCA 6 , pwCCA 7 , and CKA 8 ). Although further experiments are required, we expect our results to generalize to representations learned by (unrolled) recurrent neural network architectures 25 , 26 , if not explicitly constrained 27 . For an investigation of recurrent neural network dynamics arising from various network architectures, see Maheswaranathan et al 28 .…”

An ecologically motivated image dataset for deep learning yields better models of human vision

“…As a step in the direction of increasing the biological plausibility of deep network architectures, we here designed vNet such that the model receptive field sizes mirror the progression of foveal receptive field sizes across the human visual hierarchy. Future work should explore in how far the interplay of ecoset and the introduction of further biological details, such as recurrence ( 30 – 34 ), skip connections, and more biologically more realistic learning rules can further improve model predictions ( 6 , 8 ). Another aspect worth considering is the learning objective.…”