Reliability of Automatic Computer Vision-Based Assessment of Orofacial Kinematics for Telehealth Applications

Abstract

<b><i>Introduction:</i></b> Telehealth/remote assessment using readily available 2D mobile cameras and deep learning-based analyses is rapidly becoming a viable option for detecting orofacial and speech impairments associated with neurological and neurodegenerative disease during telehealth practice. However, the psychometric properties (e.g., internal consistency and reliability) of kinematics obtained from these systems have not been established, which is a crucial next step before their clinical usability is established. <b><i>Methods:</i></b> Participants were assessed in lab using a 3 dimensional (3D)-capable camera and at home using a readily-available 2D camera in a tablet. Orofacial kinematics was estimated from videos using a deep facial landmark tracking model. Kinematic features quantified the clinically relevant constructs of velocity, range of motion, and lateralization. In lab, all participants performed the same oromotor task. At home, participants were split into two groups that each performed a variant of the in-lab task. We quantified within-assessment consistency (Cronbach’s α), reliability (intraclass correlation coefficient [ICC]), and fitted linear mixed-effects models to at-home data to capture individual-/task-dependent longitudinal trajectories. <b><i>Results:</i></b> Both in lab and at home, Cronbach’s α was typically high (>0.80) and ICCs were often good (>0.70). The linear mixed-effect models that best fit the longitudinal data were those that accounted for individual- or task-dependent effects. <b><i>Discussion:</i></b> Remotely gathered orofacial kinematics were as internally consistent and reliable as those gathered in a controlled laboratory setting using a high-performance 3D-capable camera and could additionally capture individual- or task-dependent changes over time. These results highlight the potential of remote assessment tools as digital biomarkers of disease status and progression and demonstrate their suitability for novel telehealth applications.

Analytical Validation of a Webcam-Based Assessment of Speech Kinematics: Digital Biomarker Evaluation following the V3 Framework

“…This approach enabled us to make definitive comparisons between 2D camera-based facial tracking and gold standards; however, it is a cumbersome approach (e.g., labor-intensive to extract facial movements). Our previous work using a markerless method [31] generally supports our present findings, e.g., that facial tracking using markerless methods is reliable and can detect differences between tasks. This approach should be used in the future.…”

“…This likely reflected the well-known phenomenon of the speed-distance trade-off, in which speed increases reflect a compensation for distance and timing requirements (wider aperture but normal rate) [36]. The present results provide additional support for our previous work, in which we analyzed task-related differences in orofacial kinematics and found that markerless facial tracking methods could distinguish between oromotor tasks [31] as well as detect disease [51].…”

“…Agreement between 3D cameras and EMA has been established previously, albeit indirectly [30], and the reliability of EMA systems has also been established [34]. Our previous study also identified that 3D and 2D cameras demonstrate reliability, but the previous analysis was limited in scope [31]. Furthermore, the validity of the webcam technology for capturing expected movement properties (e.g., known differences between speaking conditions) must be established as well.…”

“…In the context of novel tool development, it is therefore important to validate consumer webcams with respect to established technologies for detecting disease-related speech impairment, e.g., EMA and 3D cameras. Previous work has established EMA systems as “gold standard” methods to capture speech-related movements [28, 29] and provided evidence of accurate performance for 3D cameras as well [30, 31].…”

Development of webcam‐collected and artificial‐intelligence‐derived social and cognitive performance measures for neurodevelopmental genetic syndromes

“…Webcam data collection also permits the frame‐by‐frame automated facial expression analysis using machine learning algorithms that enable prototype matching using large training datasets. The potential for these methods to inform neurodevelopment is strong and, increasingly, both webcam‐collected data (Simmatis et al, 2023) and artificial intelligence/machine learning algorithms (Nerusil et al, 2021) are being applied to create novel biometric measures for assessing child development and neurological conditions. A key advantage of webcam‐based data collection is that the paradigms can be administered without direct real‐time clinical supervision.…”