Repeatability of Commonly Used Speech and Language Features for Clinical Applications

Abstract

<b><i>Introduction:</i></b> Changes in speech have the potential to provide important information on the diagnosis and progression of various neurological diseases. Many researchers have relied on open-source speech features to develop algorithms for measuring speech changes in clinical populations as they are convenient and easy to use. However, the repeatability of open-source features in the context of neurological diseases has not been studied. <b><i>Methods:</i></b> We used a longitudinal sample of healthy controls, individuals with amyotrophic lateral sclerosis, and individuals with suspected frontotemporal dementia, and we evaluated the repeatability of acoustic and language features separately on these 3 data sets. <b><i>Results:</i></b> Repeatability was evaluated using intraclass correlation (ICC) and the within-subjects coefficient of variation (WSCV). In 3 sets of tasks, the median ICC were between 0.02 and 0.55, and the median WSCV were between 29 and 79%. <b><i>Conclusion:</i></b> Our results demonstrate that the repeatability of speech features extracted using open-source tool kits is low. Researchers should exercise caution when developing digital health models with open-source speech features. We provide a detailed summary of feature-by-feature repeatability results (ICC, WSCV, SE of measurement, limits of agreement for WSCV, and minimal detectable change) in the online supplementary material so that researchers may incorporate repeatability information into the models they develop.

Speech Biomarkers in Rapid Eye Movement Sleep Behavior Disorder and Parkinson Disease

“…Such inferences introduce the risk of overestimating the sensitivity of speech measures. 19 Therefore, confirmation of the stability of speech assessment from longitudinal studies over durations relevant to clinical trial time lines is essential.…”

Digital Biomarkers for Neuromuscular Disorders: A Systematic Scoping Review

“…Stegmann et al emphasized that bulbar deterioration leads to faster decline and shorter survival in ALS patients, assessed patients’ speech features digitally, and evaluated their sensitivity to detect early changes and track progression [ 15 ]. The recruited 65 ALS patients provided daily speech samples at home for three months and twice weekly for an additional six months and ALSFRS-R scores on a weekly basis.…”

“…Among diseases belonging to NMDs, ALS is one of few diseases for which a functional rating scale (ALSFRS-R) has been recognized as a gold-standard outcome measure that has been widely used in clinical practice and research [ 23 ]. One of the four studies for ALS suggested that the marker obtained through the digital sensor had a good correlation with the ALSFRS-R score [ 14 ]; another study, which assessed speech features of ALS patients, stated that the biomarkers obtained from the mobile application could detect earlier disease progression than ALSFRS-R bulbar score [ 15 ]. Although it was not a human study, an animal study showed that the index obtained through the digital sensor in SOD1gene-transgenic mice was correlated with the decline of body weight and neuromuscular function.…”

Neuropsychological test validation of speech markers of cognitive impairment in the Framingham Cognitive Aging Cohort

“…A recent study on repeatability of speech and language based features in dementia diagnosis encapsulates similar dimensions of analysis on language features (syntactic, lexical, and semantic) and demonstrates that there are features that do repeat well, evidenced by high intraclass correlations within subjects [57]. Repeatability studies are still scarce in automated speech and language analyses, and different analyses often identify different feature sets of interest.…”