On the effectiveness of facial expression recognition for evaluation of urban sound perception

Abstract

Sound perception studies mostly depend on questionnaires with fixed indicators. Therefore, it is desirable to explore methods with dynamic outputs. The present study aims to explore the effects of sound perception in the urban environment on facial expressions using software named FaceReader based on facial expression recognition (FER). The experiment involved three typical urban sound recordings, namely, traffic noise, natural sound, and community sound. A questionnaire on the evaluation of sound perception was also used, for comparison. The results show that, first, FER is an effective tool for sound perception research, since it is capable of detecting differences in participants' reactions to different sounds and how their facial expressions change over time in response to those sounds, with mean difference of valence between recordings from 0.019 to 0.059 (p<0.05or p<0.01). In a natural sound environment, for example, facial expression increased by 0.04 in the first 15 seconds and then went down steadily at 0.004 every 20 s. Second, the expression indices, namely, happy, sad, and surprised, change significantly under the effect of sound perception. In the traffic sound environment, for example, happy decreased by 0.012, sad increased by 0.032, and surprised decreased by 0.018. Furthermore, social characteristics such as distance from living place to natural environment (r=0.313), inclination to communicate (r=0.253), and preference for crowd (r=0.296) have effects on facial expression. Finally, the comparison of FER and questionnaire survey results showed that in the traffic noise recording, valence in the first 20 s best represents acoustic comfort and eventfulness; for natural sound, valence in the first 40 s best represents pleasantness; and for community sound, valence in the first 20 s of the recording best represents acoustic comfort, subjective loudness, and calmness.

The Effectiveness of Facial Expression Recognition in Detecting Emotional Responses to Sound Interventions in Older Adults With Dementia

“…In terms of individual differences, first, the point-biserial correlation analysis revealed that gender and facial expression are not significantly correlated in the music and birdsong interventions. This is consistent with previous research conclusions reached from evaluating acoustic environment using questionnaires (Meng et al, 2020a ). However, a significant correlation was found between facial expressions and gender at 20 s in the stream sound intervention, with valence significantly higher among women than men ( r = 0.869, p = 0.011).…”

“…However, in the trials with the sound of a stream and with birdsong, arousal rose again after 80 s, which may be due to distraction among the participants. The result is consistent with previous research findings (Meng et al, 2020a ). Accordingly, we chose the first 80 s as the duration for analysis in our experiment.…”

“…Leitch et al ( 2015 ) found that the length of time after tasting sweeteners affected the potency and arousal of facial expressions. In addition, Meng et al ( 2020a ) conducted laboratory experiments to test the effectiveness of facial expressions for detecting sound perception and reported that the type of sound source had a significant impact on the valence and indicators of facial expressions. FER has also been used in research on the health of older adults with dementia.…”

Noise as a Factor of Green Areas Soundscape Creation

“…Another study conducted by Meng at al. [32] using the method with dynamic results examined the impact of sound perception in an urban environment on facial expressions, measured using software called FaceReader, based on recognizing facial expressions. It can be concluded from this study that traffic noise caused mimic changes twice as fast as natural sound.…”

Deep-Emotion: Facial Expression Recognition Using Attentional Convolutional Network

“…Therefore, with the right tools, any indications preceding or following them can be subject to detection and recognition. There has been an increase in the need to detect a person's emotions in the past few years and increasing interest in human emotion recognition in various fields including, but not limited to, human-computer interfaces [1], animation [2], medicine [3,4], security [5,6], diagnostics for Autism Spectrum Disorders (ASD) in children [7], and urban sound perception [8].…”