Using adversarial networks to extend brain computer interface decoding accuracy over time

Abstract

Existing intracortical brain computer interfaces (iBCIs) transform neural activity into control signals capable of restoring movement to persons with paralysis. However, the accuracy of the "decoder" at the heart of the iBCI typically degrades over time due to turnover of recorded neurons. To compensate, decoders can be recalibrated, but this requires the user to spend extra time and effort to provide the necessary data, then learn the new dynamics. As the recorded neurons change, one can think of the underlying movement intent signal being expressed in changing coordinates. If a mapping can be computed between the different coordinate systems, it may be possible to stabilize the original decoder's mapping from brain to behavior without recalibration. We previously proposed a method based on Generalized Adversarial Networks (GANs), called "Adversarial Domain Adaptation Network" (ADAN), which aligns the distributions of latent signals within underlying low-dimensional neural manifolds. However, ADAN was tested on only a very limited dataset. Here we propose a method based on Cycle-Consistent Adversarial Networks (Cycle-GAN), which aligns the distributions of the full-dimensional neural recordings. We tested both Cycle-GAN and ADAN on data from multiple monkeys and behaviors and compared them to a linear method based on Procrustes Alignment of axes provided by Factor Analysis (PAF). Both GAN-based methods outperformed PAF. Cycle-GAN and ADAN (like PAF) are unsupervised and require little data, making them practical in real life. Overall, Cycle-GAN had the best performance and was easier to train and more robust than ADAN, making it ideal for stabilizing iBCI systems over time.

Stabilizing brain-computer interfaces through alignment of latent dynamics

“…The Day 0 dynamics model (LFADS) and Day 0 decoder (Wiener filter) were trained on a supervised dataset from a single session, and we evaluated NoMAD’s ability to enable accurate and stable decoding performance on a different session (Day K) through unsupervised alignment. We compared NoMAD against a standard Wiener filter decoder that was trained using smoothed spiking activity and behavior from Day 0 and evaluated on Day K without any adjustment ( Static decoder ), and also to two state-of-the-art manifold-based stabilization techniques: aligned factor analysis (Aligned FA) 17 , which is based on linear dimensionality reduction, and the adversarial domain adaptation network (ADAN) 18,19 , which uses a neural network autoencoder for dimensionality reduction and generative adversarial networks for alignment. We note that NoMAD alignment is completely unsupervised, in that we use all available data from a session, including periods where the monkey was inactive.…”

“…Based on data presented in Ma, et al 2022, smoothing binned spike data prior to alignment with this approach improves performance 19 . Therefore, we use 20ms binned spike data smoothed with a 40ms Gaussian kernel as input data to this method.…”

“…One promising avenue to unsupervised recalibration is the use of latent, network-level properties of neural activity [13][14][15][16] . In particular, a few recently-developed iBCIs leverage latent manifolds, revealed by the patterns of co-activation within the neuronal population, as the foundation for a more stable neural interface 9,[17][18][19] . Manifold-based iBCI decoders use a two-stage approach: first, a neurons-to-manifold mapping that transforms recorded neuronal population activity onto the underlying manifold and second, a manifold-to-behavior mapping that transforms manifold activity into intended movements [17][18][19][20] .…”

“…In particular, a few recently-developed iBCIs leverage latent manifolds, revealed by the patterns of co-activation within the neuronal population, as the foundation for a more stable neural interface 9,[17][18][19] . Manifold-based iBCI decoders use a two-stage approach: first, a neurons-to-manifold mapping that transforms recorded neuronal population activity onto the underlying manifold and second, a manifold-to-behavior mapping that transforms manifold activity into intended movements [17][18][19][20] . Because manifolds are independent of the specific neurons being recorded, different sets of recorded neurons can be mapped onto the same manifold [17][18][19][20][21][22] .…”

“…1 ). As with previous manifold decoding approaches [17][18][19]28,29 , our approach starts with a supervised training dataset containing neural activity and movement information from an initial recording session, which we call Day 0. We can use this dataset to characterize the manifold and dynamics, while also training a decoder to map manifold activity onto behavior.…”