Unsupervised Decoding of Long-Term, Naturalistic Human Neural Recordings with Automated Video and Audio Annotations

Abstract

Fully automated decoding of human activities and intentions from direct neural recordings is a tantalizing challenge in brain-computer interfacing. Implementing Brain Computer Interfaces (BCIs) outside carefully controlled experiments in laboratory settings requires adaptive and scalable strategies with minimal supervision. Here we describe an unsupervised approach to decoding neural states from naturalistic human brain recordings. We analyzed continuous, long-term electrocorticography (ECoG) data recorded over many days from the brain of subjects in a hospital room, with simultaneous audio and video recordings. We discovered coherent clusters in high-dimensional ECoG recordings using hierarchical clustering and automatically annotated them using speech and movement labels extracted from audio and video. To our knowledge, this represents the first time techniques from computer vision and speech processing have been used for natural ECoG decoding. Interpretable behaviors were decoded from ECoG data, including moving, speaking and resting; the results were assessed by comparison with manual annotation. Discovered clusters were projected back onto the brain revealing features consistent with known functional areas, opening the door to automated functional brain mapping in natural settings.

Keep it real: rethinking the primacy of experimental control in cognitive neuroscience

“…These datasets will eventually become exhausted as competing models improve and reach ceiling performance; data generators will never be out of work and there will always be a market for innovations in data acquisition. Developing technologies, such as continuous intracranial electroencephalography (iEEG; e.g., Wang et al, 2016 ), functional near-infrared spectroscopy (fNIRS; e.g., Liu et al, 2017 ), high-density diffuse optical tomography (HD-DOT; e.g., Fishell et al, 2019 ), and wearable magnetoencephalography (MEG; Boto et al, 2018 ) promise higher-fidelity and more ergonomic neuroimaging. Even the workhorse fMRI is beginning to see increased adoption of immersive virtual reality paradigms ( Mathiak and Weber, 2006 ; Spiers and Maguire, 2006 , 2007 ; Maguire, 2012 ).…”

Measuring behavior across scales

“…As our ability to record neurons in freely-behaving animals increases, the need to represent neural activity jointly with behavior is becoming increasingly apparent. As with multi-modal dynamics, most current approaches to neuro-behavioral analysis [57,68,[95][96][97][98][99][100] take a correlative or decoding approach: given one knows something about neural dynamics, what can one predict about behavior, or vice versa? This could take the form of "given a neural stimulation what did the animal do?"…”

Behavioral and Neural Variability of Naturalistic Arm Movements

“…Similar spectral power changes have also been observed in EEG and local field potential recordings across a wide variety of movement behaviors (Chung et al, 2018;Milekovic et al, 2015;Ofori et al, 2015;Tan et al, 2016). An important attribute of ECoG recordings is that the patients are being continuously monitored over long periods of time, often approximately a week, providing unique opportunities to collect long-term datasets during unconstrained, uninstructed movements (Alasfour et al, 2019;Chao et al, 2010;Gabriel et al, 2019;Vansteensel et al, 2013;Wang et al, 2018Wang et al, , 2016. However, the behavioral and neural variability of such spontaneous, naturalistic movements remains unexplored.…”