Investigating the Utility of Multimodal Conversational Technology and Audiovisual Analytic Measures for the Assessment and Monitoring of Amyotrophic Lateral Sclerosis at Scale

Abstract

We propose a cloud-based multimodal dialog platform for the remote assessment and monitoring of Amyotrophic Lateral Sclerosis (ALS) at scale. This paper presents our vision, technology setup, and an initial investigation of the efficacy of the various acoustic and visual speech metrics automatically extracted by the platform. 82 healthy controls and 54 people with ALS (pALS) were instructed to interact with the platform and completed a battery of speaking tasks designed to probe the acoustic, articulatory, phonatory, and respiratory aspects of their speech. We find that multiple acoustic (rate, duration, voicing) and visual (higher order statistics of the jaw and lip) speech metrics show statistically significant differences between controls, bulbar symptomatic and bulbar pre-symptomatic patients. We report on the sensitivity and specificity of these metrics using five-fold cross-validation. We further conducted a LASSO-LARS regression analysis to uncover the relative contributions of various acoustic and visual features in predicting the severity of patients' ALS (as measured by their self-reported ALSFRS-R scores). Our results provide encouraging evidence of the utility of automatically extracted audiovisual analytics for scalable remote patient assessment and monitoring in ALS.

Voice Analysis for Neurological Disorder Recognition–A Systematic Review and Perspective on Emerging Trends

“…The prominent Audio/Visual Emotion Challenge and Workshop ( AVEC ) addressed this aspect: featured sub-challenges in which audio and video data or features from clinical interviews ( 92 ) and interviews with virtual agents ( 93 , 94 ) from the Distress Analysis Interview Corpus [DAIC, ( 32 )] are provided as well as data on bipolar disorder ( 95 ). In addition, setups in which data is collected from the smartphone's camera as additional video input within a commercial setup are nowadays easily conceivable ( 96 ). The number of smart devices with sensors is constantly growing and therefore this topic has also been increasingly reflected in more recent dataset publications in this review ( 40 , 48 , 65 , 72 ).…”

Reliability of Automatic Computer Vision-Based Assessment of Orofacial Kinematics for Telehealth Applications

“…Remote longitudinal assessment tools for orofacial and speech applications have several clinical and research benefits. From a research perspective, these home-based assessments can be used to detect impairments associated with neurodegenerative disease that, with further development, could aid in clinical decision-making [ 15 ]. Home-based assessment also enables in-depth quantification of stability or change over time in individuals with neurological diseases due to disease progression or intervention [ 5 , 7 , 45 ].…”

“…Although these cameras are cumbersome to use and thus are not feasible for home-based use, high-quality 2D cameras in consumer electronics (e.g., smartphones and tablets) show promise for enabling high-quality orofacial assessment remotely from patients' homes. Recent research has demonstrated that that artificial intelligence-enabled remote assessment using 2D cameras can detect neurological impairment [ 15 ] and frequent at-home assessments of individuals with neurodegenerative disease can enhance disease monitoring [ 16 ]. There is an urgent need to explore the measurement properties of 2D camera systems in the context of remote assessment.…”

Analytical Validation of a Webcam-Based Assessment of Speech Kinematics: Digital Biomarker Evaluation following the V3 Framework

“…Traditional laboratory-based assessments of speech/orofacial kinematics, such as electromagnetic articulography (EMA) and 3D cameras, have a rich history of use in this research space; however, these technologies are expensive and require specialized training for operation and data analysis [16][17][18][19][20][21][22]. Consumer-grade technologies have started to show promise for disease detection/classification [4,[23][24][25]. For example, Neumann et al [24] showed that ALS could be detected across the disease severity range using laptopbased cameras and microphones in naturalistic settings via a web-based software tool.…”

“…Consumer-grade technologies have started to show promise for disease detection/classification [4,[23][24][25]. For example, Neumann et al [24] showed that ALS could be detected across the disease severity range using laptopbased cameras and microphones in naturalistic settings via a web-based software tool. These collective findings support further development of video-based approaches for digital health technologies.…”