Identifying bias in models that detect vocal fold paralysis from audio recordings using explainable machine learning and clinician ratings

Abstract

ObjectivesTo detect unilateral vocal fold paralysis (UVFP) from voice recordings using an explainable model of machine learning.Study DesignCase series - retrospective with a control group.MethodsPatients with confirmed UVFP through endoscopic examination (N=77) and controls with normal voices matched for age and sex (N=77) were included. Two tasks were used to elicit voice samples: reading the Rainbow Passage and sustaining phonation of the vowel /a/. The eighty-eight extended Geneva Minimalistic Acoustic Parameter Set (eGeMAPS) features were extracted as inputs for four machine learning models of differing complexity. Training and testing were performed using bootstrapped cross-validation. SHAP was used to identify important features.ResultsThe median Area Under the Receiver Operating Characteristic Curve (ROC AUC) score ranged from 0.79 to 0.87 depending on model and task. After removing redundant features for explainability, the highest median ROC AUC score was 0.84 using only 13 features for the vowel task and 0.87 using 39 features for the reading task. The most important features included intensity measures, mean MFCC1, mean F1 amplitude and frequency, and shimmer variability depending on model and task.ConclusionUsing the largest dataset studying UVFP to date, we achieve high performance from just a few seconds of voice recordings while discovering which acoustic features are important across models. Notably, we demonstrate that the models use different combinations of features to achieve similar effect sizes. Overall the categories of features related to vocal fold physiology were conserved across the models. Machine learning thus provides a mechanism to detect UVFP and contextualize the accuracy relative to both model architecture and pathophysiology.Level of EvidenceType 3

Recognizing Edge-Based Diseases of Vocal Cords by Using Convolutional Neural Networks

“…Zhao et al proposed a hybrid method involving the application of a convolutional neural network and transfer learning for the classification of VC lesions [47]. Low et al compared the performance of several machine learning methods (e.g., logistic regression, random forest, and the Stochastic Gradient Descent classifier) in detecting unilateral vocal fold paralysis [48]. To trace the sound generation and evaluate the health of VCs, Yousef et al applied an unsupervised machine-learning method and an active contour modeling technique to identify the position of the glottis to understand VC actions [49].…”