What is changing when: Decoding visual information in movies from human intracranial recordings

Abstract

The majority of visual recognition studies have focused on the neural responses to repeated presentations of static stimuli with abrupt and well-defined onset and offset times. In contrast, natural vision involves unique renderings of visual inputs that are continuously changing without explicitly defined temporal transitions. Here we considered commercial movies as a coarse proxy to natural vision. We recorded intracranial field potential signals from 1,284 electrodes implanted in 15 patients with epilepsy while the subjects passively viewed commercial movies. We could rapidly detect large changes in the visual inputs within approximately 100 ms of their occurrence, using exclusively field potential signals from ventral visual cortical areas including the inferior temporal gyrus and inferior occipital gyrus. Furthermore, we could decode the content of those visual changes even in a single movie presentation, generalizing across the wide range of transformations present in a movie. These results present a methodological framework for studying cognition during dynamic and natural vision.

Neurons detect cognitive boundaries to structure episodic memories in humans

“…5d-h). We hypothesize that the early responses of boundary cells (mostly in the parahippocampal gyrus) reflect contextual changes detected in the higher-level visual areas 27,28 , while event cells (mostly in the hippocampus) are the result of a late output signal from a comparator operation 29,30 (between predicted and received signal).…”

“…This suggests that besides the early HB-related responses, there is a secondary later response to HBs that is encoded as a distributed population response. We hypothesize that the early responses of boundary cells reflect contextual changes detected in the higher-level visual areas 33 , 34 , while event cells are the result of a late output signal from a comparator operation between predicted and received signals 35 , 36 .…”

Semantic novelty modulates neural responses to visual change across the human brain

“…However, we also find a large number of channels with specific responses to event cuts widely distributed across the brain outside of the medial temporal lobe (Figure 3C&D). Other work with intracranial recordings in humans has been able to decode scene identity from recording locations widespread across the whole temporal lobe (5). Specific responses to event salience have also been identified in the orbitofrontal cortex (23).…”

“…It is a more powerful driver of neural responses than low-level features such as luminance and contrast (3, 4). Visual motion is also caused by eye movements, which many studies consider to be a confound (5, 6). Complicating matters, motion in movies can also attract and guide eye movements (7).…”

Growing evidence for separate neural mechanisms for attention and consciousness

“…First, thanks to the success of deep brain stimulation, there is an ever-increasing number of people who live for extended periods of time with chronic electrode implants in their brains. This development goes hand-in-hand with invasive neurophysiological recordings that provide resolution at the level of single neurons becoming increasingly commonplace in human neurological patients (Aquino et al, 2020;Isik, Singer, Madsen, Kanwisher, & Kreiman, 2018;Kreiman, Koch, & Fried, 2000;Pryluk, Kfir, Gelbard-Sagiv, Fried, & Paz, 2019;Rees, Kreiman, & Koch, 2002;Rutishauser et al, 2013;Rutishauser et al, 2015). Given this trend, it seems increasingly likely that we are facing a future where neurophysiological implants are used in healthy individuals (Musk, 2019).…”