A Generalizable Speech Emotion Recognition Model Reveals Depression and Remission

Abstract

Objective: Affective disorders have long been associated with atypical voice patterns, however, current work on automated voice analysis often suffers from small sample sizes and untested generalizability. This study investigated a generalizable approach to aid clinical evaluation of depression and remission from voice. Methods: A Mixture-of-Experts machine learning model was trained to infer happy/sad emotional state using three publicly available emotional speech corpora. We examined the model's predictive ability to classify the presence of depression on Danish speaking healthy controls (N = 42), patients with first-episode major depressive disorder (MDD) (N = 40), and the same patients in remission (N = 25) based on recorded clinical interviews. The model was evaluated on raw data, data cleaned for background noise, and speaker diarized data. Results: The model showed reliable separation between healthy controls and depressed patients at the first visit, obtaining an AUC of 0.71. Further, we observed a reliable treatment effect in the depression group, with speech from patients in remission being indistinguishable from that of the control group. Model predictions were stable throughout the interview, suggesting that as little as 20-30 seconds of speech is enough to accurately screen a patient. Background noise (but not speaker diarization) heavily impacted predictions, suggesting that a controlled environment and consistent preprocessing pipelines are crucial for correct characterizations. Conclusion: A generalizable speech emotion recognition model can effectively reveal changes in speaker depressive states before and after treatment in patients with MDD. Data collection settings and data cleaning are crucial when considering automated voice analysis for clinical purposes.

Vocal markers of autism: assessing the generalizability of machine learning models

“…The 5 SVM models were used to assess the test sets, and their predictions were combined into a single voting ensemble (Brownlee, 2020; Hansen et al, 2021; Sechidis et al, 2021). Each model made a prediction for each voice recording in the test set, and the ensemble model gave a final predicted class based on the majority of these model votes.…”

“…Each model made a prediction for each voice recording in the test set, and the ensemble model gave a final predicted class based on the majority of these model votes. Note that other systems beyond majority rules have been developed, e.g., Mixture of Experts with weights based on similarity between test and training data (Hansen et al, 2021; Sechidis et al, 2021). Combining or utilizing multiple models within a single model – such as an ensemble model – benefits performance and generalizability, since no two models are likely to overfit in the same way and different models can compensate for each other ‘s biases (Buracas & Albright, 1993; Hong & Page, 2004; Tang et al, 2005).…”

“…Each model made a prediction for each voice recording in the test set, and the ensemble model gave a final predicted class based on the majority of these model votes. Note that other systems beyond majority rules have been developed, e.g., Mixture of Experts with weights based on similarity between test and training data (Hansen et al, 2021;Sechidis et al, 2021).…”

“…If a new language were to become available, a new model could be trained only on the new data and added to the mixture. This approach has already shown promising results in related fields, predicting the emotional content of Parkinson's and depressive speech in languages on which the models had not been trained (Hansen et al, 2021;Sechidis et al, 2021). Analogously modular approaches have been developed for language models, including novel techniques to mix, re-weight, add or entirely remove "experts" according to the task at hand (Gururangan et al, 2021).…”

“…Since exploratory analyses showed little ability of the models to generalize to female voices, but female participants were too few and unbalanced across datasets to be able to draw any conclusion, we excluded them from the current study, and therefore the table. For a more detailed analysis of differences in vocal markers of autism depending on biological sex see Fusaroli et al (2021).…”