Multimodal explainable AI predicts upcoming speech behavior in adults who stutter

Abstract

A key goal of cognitive neuroscience is to better understand how dynamic brain activity relates to behavior. Such dynamics, in terms of spatial and temporal patterns of brain activity, are directly measured with neurophysiological methods such as EEG, but can also be indirectly expressed by the body. Autonomic nervous system activity is the best-known example, but, muscles in the eyes and face can also index brain activity. Mostly parallel lines of artificial intelligence research show that EEG and facial muscles both encode information about emotion, pain, attention, and social interactions, among other topics. In this study, we examined adults who stutter (AWS) to understand the relations between dynamic brain and facial muscle activity and predictions about future behavior (fluent or stuttered speech). AWS can provide insight into brain-behavior dynamics because they naturally fluctuate between episodes of fluent and stuttered speech behavior. We focused on the period when speech preparation occurs, and used EEG and facial muscle activity measured from video to predict whether the upcoming speech would be fluent or stuttered. An explainable self-supervised multimodal architecture learned the temporal dynamics of both EEG and facial muscle movements during speech preparation in AWS, and predicted fluent or stuttered speech at 80.8% accuracy (chance=50%). Specific EEG and facial muscle signals distinguished fluent and stuttered trials, and systematically varied from early to late speech preparation time periods. The self-supervised architecture successfully identified multimodal activity that predicted upcoming behavior on a trial-by-trial basis. This approach could be applied to understanding the neural mechanisms driving variable behavior and symptoms in a wide range of neurological and psychiatric disorders. The combination of direct measures of neural activity and simple video data may be applied to developing technologies that estimate brain state from subtle bodily signals.

Applications of Self-Supervised Learning to Biomedical Signals: where are we now

“…In the medical domain, multimodal data are often complementary with each other, meaning that each type of data (e.g. images, sensor data, text report) can be used to extract unique latent representations to allow a better understanding of a pathology [160] 2021 Predictive Seizure Forecasting Private 3 Saeed et al [161] 2021 Multiple Multiple HHAR [144], MobiAct [147], MotionSense [149], UCI HAR [146], HAPT [162], Sleep-EDF [131], MIT Driver DB [163], WiFi CSI [164] 4 Spathis et al [165] 2021 Predictive (HR forecasting) Subject Health Private 5 Thiam et al [166] 2021 Generative Pain Classification BioVid heat pain [167], SenseEmotion [168] 6 Das et al [169] 2022 Predictive Stuttering prediction Private 7 Deldari et al [170] 2022 Contrastive (COCOA) Multiple UCI HAR [146], SLEEP-EDF, PAMAP2 [171], WE-SAD [87], Opportunity [172] 8 Lemkhenter et al [173] 2022 Predictive (PhaseSwap)…”

“…The last two selected multimodal approaches combine biosignals with video recordings. Leveraging a combination of EEG and facial activity data extracted from video, Das et al [169] trained an explainable AI model to predict upcoming speech stuttering. Also, Martini et al [160] showed the potentiality of multimodal self-supervised learning by combining stereoencephalography (SEEG) and video data to forecast seizure events in drug resistant epileptic subjects.…”

Applications of Self-Supervised Learning to Biomedical Signals: A Survey

“…Leveraging a combination of EEG and facial activity data extracted from video, Das et al . [185] trained an explainable AI model to predict upcoming speech stuttering, while Martini et al . [178] showed the potentiality of multimodal selfsupervised learning by combining stereoencephalography (SEEG) and video data to forecast seizure events in drug resistant epileptic subjects.…”

From User Perceptions to Technical Improvement: Enabling People Who Stutter to Better Use Speech Recognition

“…2.1.3 Dysfluent Speech Recognition. Technical work on improving speech assistants for PWS has focused on ASR models [8,23,31,35,50,51,61], stuttering detection [43], dysfluency detection or classification [22,40,42,48,56], clinical assessment [11], and dataset development [12,37,42,55]. Shonibare et al [61] and Mendelev et al [50] investigate training end-to-end RNN-T ASR models on speech from PWS.…”

“…Research on speech technology for PWS has largely focused on technical improvements to automatic speech recognition (ASR) models [31,35,50,51,61], dysfluency detection [22,40,42,48], and dataset development [12,37,42,55]. This body of work has largely lacked a human-centered approach to understanding the experiences that PWS have with speech recognition systems [17], which could in turn inform how to prioritize and advance technical improvements.…”