Validation and application of the Non-Verbal Behavior Analyzer: An automated tool to assess non-verbal emotional expressions in psychotherapy

Abstract

BackgroundEmotions play a key role in psychotherapy. However, a problem with examining emotional states via self-report questionnaires is that the assessment usually takes place after the actual emotion has been experienced which might lead to biases and continuous human ratings are time and cost intensive. Using the AI-based software package Non-Verbal Behavior Analyzer (NOVA), video-based emotion recognition of arousal and valence can be applied in naturalistic psychotherapeutic settings. In this study, four emotion recognition models (ERM) each based on specific feature sets (facial: OpenFace, OpenFace-Aureg; body: OpenPose-Activation, OpenPose-Energy) were developed and compared in their ability to predict arousal and valence scores correlated to PANAS emotion scores and processes of change (interpersonal experience, coping experience, affective experience) as well as symptoms (depression and anxiety in HSCL-11).Materials and methodsA total of 183 patient therapy videos were divided into a training sample (55 patients), a test sample (50 patients), and a holdout sample (78 patients). The best ERM was selected for further analyses. Then, ERM based arousal and valence scores were correlated with patient and therapist estimates of emotions and processes of change. Furthermore, using regression models arousal and valence were examined as predictors of symptom severity in depression and anxiety.ResultsThe ERM based on OpenFace produced the best agreement to the human coder rating. Arousal and valence correlated significantly with therapists’ ratings of sadness, shame, anxiety, and relaxation, but not with the patient ratings of their own emotions. Furthermore, a significant negative correlation indicates that negative valence was associated with higher affective experience. Negative valence was found to significantly predict higher anxiety but not depression scores.ConclusionThis study shows that emotion recognition with NOVA can be used to generate ERMs associated with patient emotions, affective experiences and symptoms. Nevertheless, limitations were obvious. It seems necessary to improve the ERMs using larger databases of sessions and the validity of ERMs needs to be further investigated in different samples and different applications. Furthermore, future research should take ERMs to identify emotional synchrony between patient and therapists into account.

Automatisierte Analysen von Psychotherapievideos

“…Neben der reinen Quantität an Bewegung können auch spezifische Bewegungen erfasst werden. Ein noch präziserer Ansatz erfasst einzelne relevante Körperpunkte und kann somit Muskelbewegungen erkennen, um u. a. Emotionen im Gesichtsausdruck zu identifizieren 2 . Was verbal, also inhaltlich, in der Therapiesitzung gesprochen wurde, kann durch Transkription, also die Übertragung der auditiven Informationen in verschriftlichte Texte, aufbereitet werden 3 .…”

The increasing instance of negative emotion reduce the performance of emotion recognition

“…From this perspective, recognizing negative emotions is extremely important for affective HCI (Lee and Yoo, 2020). Research on negative emotions can help guide the diagnosis and treatment of mood disorders (Terhürne et al, 2022;Wu et al, 2022). Typical application scenarios for negative emotion detection include the online mental healthcare platforms or an indicator to assess the quality of call center conversations (Vaudable and Devillers, 2012;Gouizi et al, 2014;Dheeraj and Ramakrishnudu, 2021).…”

Digitalisierung und maschinelles Lernen in der Psychotherapieforschung und Praxis – Potentiale und Probleme

“…und Gesichtsdaten [10], sowie physiologischen Parametern [11]. Darüber hinaus wurde ebenfalls das Potential von EMA und der passiven Datenerhebung via Smartphones zur Vorhersage von Psychotherapieerfolg [12,13] und Therapieabbruch [14] ersichtlich.…”

“…Erste Forschungsergebnisse zur Integration von ML und multimodalen Datensätzen zeigen vielversprechende Befunde hinsichtlich der Vorhersage von psychischen Erkrankungen. Dazu zählen erste Studien zur Nützlichkeit von Sprach- 9 und Gesichtsdaten 10 , sowie physiologischen Parametern 11 . Darüber hinaus wurde ebenfalls das Potential von EMA und der passiven Datenerhebung via Smartphones zur Vorhersage von Psychotherapieerfolg 12 13 und Therapieabbruch 14 ersichtlich.…”