A Self-Supervised Deep Neural Network for Image Completion Resembles Early Visual Cortex fMRI Activity Patterns for Occluded Scenes

Abstract

The promise of artificial intelligence in understanding biological vision relies on the comparison of computational models with brain data that captures functional principles of visual information processing. Deep neural networks (DNN) have successfully matched the transformations in hierarchical processing occurring along the brain's feedforward visual pathway extending into ventral temporal cortex. However, we are still to learn if DNNs can successfully describe feedback processes in early visual cortex. Here, we investigated similarities between human early visual cortex and a DNN with encoder/decoder architecture, trained in an unsupervised fashion to fill occlusions and reconstruct an unseen image. Using Representational Similarity Analysis (RSA), we compared 3T fMRI data from a non-stimulated patch of early visual cortex in human participants viewing partially occluded images, with the different DNN layer activations from the same images. Results show that our network outperforms a classical supervised network (VGG16) in terms of similarity to fMRI data, meaning that improved neural network models of vision need to incorporate architectures capturing cortical feedback processing. We also find that DNN decoder pathway activations are more similar to brain processing compared to encoder activations, suggesting an integration of mid-and low/middle-level features in early visual cortex. Challenging an AI model and the human brain to solve the same task offers a valuable way to compare DNNs with brain data and helps to constrain our understanding of information processing such as neuronal predictive coding.Keywords Unsupervised deep neural networks · Encoder/decoder architecture · 3T fMRI · Early visual cortex · Representational Similarity Analysis (RSA).

A brain-inspired object-based attention network for multi-object recognition and visual reasoning

“…The generative nature of this method makes it suitable for modeling different top-down modulations and feedback processing. To date, these models have been used to study effects of top-down feedback in ventral pathway [76, 2] and to model predictive coding [37], mental imagery [7] and continual learning [83]. More generally, these models can also be used for representation learning, where they can be trained using self-supervised methods to generate the visual input.…”