Evaluating the Impact of Voice Activity Detection on Speech Emotion Recognition for Autistic Children

Abstract

Individuals with autism are known to face challenges with emotion regulation, and express their affective states in a variety of ways. With this in mind, an increasing amount of research on automatic affect recognition from speech and other modalities has recently been presented to assist and provide support, as well as to improve understanding of autistic individuals' behaviours. As well as the emotion expressed from the voice, for autistic children the dynamics of verbal speech can be inconsistent and vary greatly amongst individuals. The current contribution outlines a voice activity detection (VAD) system specifically adapted to autistic children's vocalisations. The presented VAD system is a recurrent neural network (RNN) with long short-term memory (LSTM) cells. It is trained on 130 acoustic Low-Level Descriptors (LLDs) extracted from more than 17 h of audio recordings, which were richly annotated by experts in terms of perceived emotion as well as occurrence and type of vocalisations. The data consist of 25 English-speaking autistic children undertaking a structured, partly robot-assisted emotion-training activity and was collected as part of the DE-ENIGMA project. The VAD system is further utilised as a preprocessing step for a continuous speech emotion recognition (SER) task aiming to minimise the effects of potential confounding information, such as noise, silence, or non-child vocalisation. Its impact on the SER performance is compared to the impact of other VAD systems, including a general VAD system trained from the same data set, an out-of-the-box Web Real-Time Communication (WebRTC) VAD system, as well as the expert annotations. Our experiments show that the child VAD system achieves a lower performance than our general VAD system, trained under identical conditions, as we obtain receiver operating characteristic area under the curve (ROC-AUC) metrics of 0.662 and 0.850, respectively. The SER results show varying performances across valence and arousal depending on the utilised VAD system with a maximum concordance correlation coefficient (CCC) of 0.263 and a minimum root mean square error (RMSE) of 0.107. Although the performance of the SER models is generally low, the child VAD system can lead to slightly improved results compared to other VAD systems and in particular the VAD-less baseline, supporting the hypothesised importance of child VAD systems in the discussed context.

Investigating Automatic Speech Emotion Recognition for Children with Autism Spectrum Disorder in interactive intervention sessions with the social robot Kaspar

“…As suggested by Milling et al (2022), the performance of VAD systems can have an impact on the estimation of affective states based on audio recordings. This hypothesis is supported by the clear differences in valence and arousal predictions in Table 5, when comparing the average absolute difference between VAD-based SER predictions and the non-VAD-based SER predictions, in particular for the arousal case.…”

“…The model was trained on audio data from the British study arm of the project containing audio data from 84 robot-supported intervention sessions of 25 English-speaking children with ASD from the United Kingdom, having a total duration of more than 17 hours. For the experiments reported by Milling et al (2022), the data was partitioned into train, development, and test partitions in a speaker-independent way. In total, this data contains more than 2 hours of annotated child vocalisations (partially overlapping with other vocalisations).…”

“…Finally, the VAD system summarises frame-level predictions with a detection event if the EER threshold is reached for at least 0.25 s of a 1-s-chunk. More details on the VAD are given in Milling et al (2022). The child VAD system has a reported EER of 0.381 and an area-under-the-curve of 0.662 on the DE-ENIGMA corpus.…”

“…In order to estimate emotions from speech we use a deep learning model developed, trained, and evaluated by Milling et al (2022), which is again based on the same DE-ENIGMA data and partitioning of British children, as described for the VAD component. The models are designed for a continuous speech emotion recognition (SER) task and we apply correspondingly trained models for two types of tasks: 1) each 1 s chunk from the original audio recording is used for the continuous SER task and 2) only 1 s chunks, for which a child vocalisation was detected, are being used for the continuous SER task.…”

“…Reported performances of the models achieve up to 0.168 concordance correlation coefficient (CCC) on the test partition of the DE-ENIGMA corpus. More details on the SER system and performance values can be found in Milling et al (2022).…”