Investigating naturalistic hand movements by behavior mining in long-term video and neural recordings

Singh, Satpreet H.; Peterson, Steven M.; N., Rao, Rajesh P.; Brunton, Bingni W.

doi:10.48550/arxiv.2001.08349

Cited by 1 publication

(2 citation statements)

References 0 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Through a series of systematic experiments, we use decoders tailored to each participant-by far the most common approach to training decodersas a standard to which the performance of our generalized decoders are compared (figure 1(B)). In particular, our training data are from 12 ECoG participants during uninstructed, naturalistic arm movements (figure 1(C), [57,38]); our test data are then either one ECoG participant withheld from the training set, or participants from an entirely independent EEG dataset. HTNet consistently outperformed other decoders, and fine-tuning pre-trained HTNet decoders with a small number of the unseen participant's events yielded decoders that approached the performance of tailored decoders trained on many more events.…”

Section: Resultsmentioning

confidence: 99%

“…Our decoding task was to classify upper-limb 'move' and 'rest' events of the arm contralateral to the implanted electrode hemisphere. We obtained non-concurrent move and rest events from video recordings via markerless pose tracking and automated state segmentation (see Singh et al [57] for further details). Move events correspond to wrist movement that occurred after at least 0.5 s of no movement, while rest events indicate no movement in either wrist for at least 3 s.…”

Section: Intracranial Electrocorticography (Ecog) Datasetmentioning

confidence: 99%

See 1 more Smart Citation

Generalized neural decoders for transfer learning across participants and recording modalities

et al. 2021

View full text Add to dashboard Cite

Objective: Advances in neural decoding have enabled brain-computer interfaces to perform increasingly complex and clinically-relevant tasks. However, such decoders are often tailored to specific participants, days, and recording sites, limiting their practical long-term usage. Therefore, a fundamental challenge is to develop neural decoders that can robustly train on pooled, multi-participant data and generalize to new participants.Approach: We introduce a new decoder, HTNet, which uses a convolutional neural network with two innovations: (1) a Hilbert transform that computes spectral power at data-driven frequencies and (2) a layer that projects electrodelevel data onto predefined brain regions. The projection layer critically enables applications with intracranial electrocorticography (ECoG), where electrode locations are not standardized and vary widely across participants. We trained HTNet to decode arm movements using pooled ECoG data from 11 of 12 participants and tested performance on unseen ECoG or electroencephalography (EEG) participants; these pretrained models were also subsequently fine-tuned to each test participant.Main results: HTNet outperformed state-of-the-art decoders when tested on unseen participants, even when a different recording modality was used. By finetuning these generalized HTNet decoders, we achieved performance approaching the best tailored decoders with as few as 50 ECoG or 20 EEG events. We were also able to interpret HTNet's trained weights and demonstrate its ability to extract physiologically-relevant features.Significance: By generalizing to new participants and recording modalities, robustly handling variations in electrode placement, and allowing participantspecific fine-tuning with minimal data, HTNet is applicable across a broader range of neural decoding applications compared to current state-of-the-art decoders.

show abstract

Section: Resultsmentioning

confidence: 99%

Section: Intracranial Electrocorticography (Ecog) Datasetmentioning

confidence: 99%