Decoding of human identity by computer vision and neuronal vision

Abstract

Extracting meaning from a dynamic and variable flow of incoming information is a major goal of both natural and artificial intelligence. Computer vision (CV) guided by deep learning (DL) has made significant strides in recognizing a specific identity despite highly variable attributes. This is the same challenge faced by the nervous system and partially addressed by the concept cells—neurons exhibiting selective firing in response to specific persons/places, described in the human medial temporal lobe (MTL) ⁠. Yet, access to neurons representing a particular concept is limited due to these neurons’ sparse coding. It is conceivable, however, that the information required for such decoding is present in relatively small neuronal populations. To evaluate how well neuronal populations encode identity information in natural settings, we recorded neuronal activity from multiple brain regions of nine neurosurgical epilepsy patients implanted with depth electrodes, while the subjects watched an episode of the TV series “24”. First, we devised a minimally supervised CV algorithm (with comparable performance against manually-labeled data) to detect the most prevalent characters (above 1% overall appearance) in each frame. Next, we implemented DL models that used the time-varying population neural data as inputs and decoded the visual presence of the four main characters throughout the episode. This methodology allowed us to compare “computer vision” with “neuronal vision”—footprints associated with each character present in the activity of a subset of neurons—and identify the brain regions that contributed to this decoding process. We then tested the DL models during a recognition memory task following movie viewing where subjects were asked to recognize clip segments from the presented episode. DL model activations were not only modulated by the presence of the corresponding characters but also by participants’ subjective memory of whether they had seen the clip segment, and by the associative strengths of the characters in the narrative plot. The described approach can offer novel ways to probe the representation of concepts in time-evolving dynamic behavioral tasks. Further, the results suggest that the information required to robustly decode concepts is present in the population activity of only tens of neurons even in brain regions beyond MTL.

Decoding movie content from neuronal population activity in the human medial temporal lobe

“…In related research by Zhang et al 19 , where temporal distance was not accounted for, high decoding performances were reported on a similar task. Based on our analyses (Fig.…”

“…S8). This might explain the higher decoding performance for a comparable task in Zhang et al 19 , which did not report the use of temporal gaps for model evaluation. All subsequently reported results refer to the performance on the held-out test data using 5-fold cross-validation, with data splits incorporating the most conservative temporal gap of 32 s (see Methods).…”

“…Much of our current understanding of semantic representations in the brain, and specifically concept cells, has been established from static presentations of semantic information, and it is currently unclear to what extent these insights translate to natural environments. Previous work using video stimuli has revealed the presence of cells sensitive to event boundaries [13][14][15][16] , stimulus-driven inter-subject correlations of neural activity 17,18 , single-trial signatures of movie content 16 , and character-related population activity 19 . While such work has partially addressed which semantic features of a naturalistic, continuous stimulus can be reliably decoded from the population activity of single neurons within the human MTL, there is presently no direct comparison between the representation of semantic information on the single-neuron level versus the population level under dynamic viewing conditions.…”

Gradient-Guided Modality Decoupling for Missing-Modality Robustness

“…Multimodal learning In multimodal learning, different modalities, e.g., texts, images (Zhang et al 2023), videos (Zadeh et al 2016(Zadeh et al , 2018, Magnetic Resonance Image (MRI) modalities (Dinsdale et al 2021) or other sources of information (Lu et al 2022;Zhang et al 2022), are often considered to be complementary to each other. Fusion has been widely studied as one of the most critical topics in multimodal learning.…”

Decoding region-level visual functions from invasive EEG data

“…The potential to reconstruct visual information from the brain may rely on discerning specific neuronal spike patterns in familiar environments, facilitating the transformation of spatio-temporal features into meaningful images. While diverse architectures of artificial neural network (ANN)-based machines have been proposed, interpretability plays a crucial role in determining their adequacy as brain models [8, 9, 10, 11, 12, 13]. Hence, it is important not only to improve the accuracy of visual decoding but also to reveal the underlying mechanism in the mapping from visual stimuli to brain activity.…”