NeuroVAD: Real-Time Voice Activity Detection from Non-Invasive Neuromagnetic Signals

Abstract

Neural speech decoding-driven brain-computer interface (BCI) or speech-BCI is a novel paradigm for exploring communication restoration for locked-in (fully paralyzed but aware) patients. Speech-BCIs aim to map a direct transformation from neural signals to text or speech, which has the potential for a higher communication rate than the current BCIs. Although recent progress has demonstrated the potential of speech-BCIs from either invasive or non-invasive neural signals, the majority of the systems developed so far still assume knowing the onset and offset of the speech utterances within the continuous neural recordings. This lack of real-time voice/speech activity detection (VAD) is a current obstacle for future applications of neural speech decoding wherein BCI users can have a continuous conversation with other speakers. To address this issue, in this study, we attempted to automatically detect the voice/speech activity directly from the neural signals recorded using magnetoencephalography (MEG). First, we classified the whole segments of pre-speech, speech, and post-speech in the neural signals using a support vector machine (SVM). Second, for continuous prediction, we used a long short-term memory-recurrent neural network (LSTM-RNN) to efficiently decode the voice activity at each time point via its sequential pattern-learning mechanism. Experimental results demonstrated the possibility of real-time VAD directly from the non-invasive neural signals with about 88% accuracy.

MEG Sensor Selection for Neural Speech Decoding

“…The classification was performed with a 5-fold cross-validation (CV) strategy using a 2 nd order polynomial kernel. The choice of this 2 nd order kernel was motivated from our prior work [ 45 ], [ 70 ] which found that this kernel performs optimally compared to other kernels (radial basis function (RBF), sigmoid, 3 rd and 4 th order polynomial) for this MEG data. C parameter tells the SVM optimization how much is needed to avoid misclassifying each training example.…”

“…The classification was performed with a 5-fold cross-validation (CV) strategy using a 2 nd order polynomial kernel. The choice of this 2 nd order kernel was motivated from our prior work [46], [71] which found that this kernel performs optimally compared to other FIGURE 3: Forward sensor selection algorithm: This stepwise selection algorithm selects the most optimal set of sensors starting from 1 to 50, one by one in each step. First, the algorithm selects the first optimal sensor (O 1 ) which results in best CV accuracy when the SVM was trained with each sensor as input for 196 times (total number of sensors = 196).…”

“…These studies provide strong support for the efficacy of using MEG for speech-based BCI studies. Specifically, the advantages in the high temporal resolution of MEG has been proven to be effective by several MEG studies including the superiority of MEG recordings for speech production [41], neural tracking of speech envelope [42], investigating temporal dynamics of communicative language processing (naming and requesting) [43], temporal patterns of neural activations in speech [44], automatic neural speech activity recognition [45], and realtime voice activity detection from neuromagnetic signals (NeuroVAD) [46]. These studies provide strong evidence for the ability of MEG to decode speech from neuromagnetic signals.…”

“…We chose a support vector machine (SVM) decoder for classification considering its faster training ability with small datasets and well suitability when combined with feature selection algorithms [ 58 ]. Moreover, SVMs have been proven to be effective in various classification tasks including our previous works with MEG [ 45 ], [ 59 ]. In addition, we also investigated a stacked autoencoder-based dimension reduction strategy to reconfirm the improved decoding accuracy.…”

Phonocardiogram Signal Processing for Automatic Diagnosis of Congenital Heart Disorders through Fusion of Temporal and Cepstral Features

“…Features are extracted from each class (normal, ASD, VSD). The SVM classifier is a widely applied method of classification for biomedical signals [ 62 , 63 , 64 , 65 ] due to its excellent generalization capability. It obtains the optimal separating hyperplane for class separation by converting input features to higher dimensions through some nonlinear mapping [ 66 ].…”