Decoding Pixel-Level Image Features From Two-Photon Calcium Signals of Macaque Visual Cortex

Abstract

Images of visual scenes comprise essential features important for visual cognition of the brain. The complexity of visual features lies at different levels, from simple artificial patterns to natural images with different scenes. It has been a focus of using stimulus images to predict neural responses. However, it remains unclear how to extract features from neuronal responses. Here we address this question by leveraging two-photon calcium neural data recorded from the visual cortex of awake macaque monkeys. With stimuli including various categories of artificial patterns and diverse scenes of natural images, we employed a deep neural network decoder inspired by image segmentation technique. Consistent with the notation of sparse coding for natural images, a few neurons with stronger responses dominated the decoding performance, whereas decoding of ar tificial patterns needs a large number of neurons. When natural images using the model pretrained on artificial patterns are decoded, salient features of natural scenes can be extracted, as well as the conventional category information. Altogether, our results give a new perspective on studying neural encoding principles using reverse-engineering decoding strategies.

Deep learning-based location decoding reveals that across-day representational drift is better predicted by rewarded experience than time

“…Our general framework for quantifying representational drift is versatile. By defining drift as a measure of the degradation of quality of a deep neural decoder trained on wavelet images, our exact method can be applied to any brain region, and any representation which can be decoded; calcium imaging data, for example, has been shown to be well suited to decoding using deep networks 21,32 . In this way our method can facilitate research into the drifting behaviours of a wide variety of neural representations and functionalities.…”

Decoding dynamic visual scenes across the brain hierarchy

“…While the interaction between encoding and decoding has been explored for simple stimuli such as directional selectivity [55], the investigation of complex natural scenes has been limited due to constraints in computational models and methodologies [22], [24]. Previous studies have shown that DNNs can be valuable tools for decoding and reconstructing pixel-level information of natural scenes [43], [44], [47], [50], yet these studies lacked a clear correlation with established encoding metrics. Our present work aims to elucidate this tight correlation, enabling the use of DNNs to quantify neuronal response patterns and compute the differences between patterns in response to both artificial and natural stimuli.…”

Neural Decoding of Visual Information Across Different Neural Recording Modalities and Approaches

“…Transfer learning is an emerging technique in machine learning [112,113] in which a machine exploits the knowledge gained from a previous task to improve the generalizability of another [114,115] ; usually, the scale of data in the original training task is larger than that in the new problem. In this case, the advantages of transfer learning are obvious, i.e., it reduces training time, results in better neural network performance (in most cases), and does not need much data.…”