Opportunities and Challenges for Using Automatic Human Affect Analysis in Consumer Research

Küster, Dennis; Krumhuber, Eva; Steinert, Lars; Ahuja, Anuj; Baker, Marc; Schultz, Tanja

doi:10.3389/fnins.2020.00400

Cited by 16 publications

(9 citation statements)

References 67 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…In the wake of rapid advances in computer vision and machine learning, competing computational approaches now exist that focus on the analysis of facial expressions. Automatic facial affect recognition has significant advantages in terms of time and labor costs over human coding [22] and has been envisioned to give rise to numerous applications in fields as diverse as security, medicine, education, telecommunication, automotive, and marketing industries [23,24]. While the computational modelling of emotional expressions forms a narrow, although increasingly common, approach, the ultimate aim is to build human-computer interfaces that not only detect but also respond to emotional signals of the user [25,26].…”

Section: Introductionmentioning

confidence: 99%

A performance comparison of eight commercially available automatic classifiers for facial affect recognition

et al. 2020

Self Cite

View full text Add to dashboard Cite

In the wake of rapid advances in automatic affect analysis, commercial automatic classifiers for facial affect recognition have attracted considerable attention in recent years. While several options now exist to analyze dynamic video data, less is known about the relative performance of these classifiers, in particular when facial expressions are spontaneous rather than posed. In the present work, we tested eight out-of-the-box automatic classifiers, and compared their emotion recognition performance to that of human observers. A total of 937 videos were sampled from two large databases that conveyed the basic six emotions (happiness, sadness, anger, fear, surprise, and disgust) either in posed (BU-4DFE) or spontaneous (UT-Dallas) form. Results revealed a recognition advantage for human observers over automatic classification. Among the eight classifiers, there was considerable variance in recognition accuracy ranging from 48% to 62%. Subsequent analyses per type of expression revealed that performance by the two best performing classifiers approximated those of human observers, suggesting high agreement for posed expressions. However, classification accuracy was consistently lower (although above chance level) for spontaneous affective behavior. The findings indicate potential shortcomings of existing out-of-thebox classifiers for measuring emotions, and highlight the need for more spontaneous facial databases that can act as a benchmark in the training and testing of automatic emotion recognition systems. We further discuss some limitations of analyzing facial expressions that have been recorded in controlled environments.

show abstract

Section: Introductionmentioning

confidence: 99%

A performance comparison of eight commercially available automatic classifiers for facial affect recognition

et al. 2020

Self Cite

View full text Add to dashboard Cite

show abstract

“…Given the various methods employed for eliciting and validating dynamic facial expressions, the quantity and quality of data available on emotion recognition performance is a major issue (Küster et al, 2020 ). There is currently no normative standard that incorporates the diversity of approaches seen in the literature.…”

Section: Introductionmentioning

confidence: 99%

Human and machine validation of 14 databases of dynamic facial expressions

et al. 2020

Self Cite

View full text Add to dashboard Cite

With a shift in interest toward dynamic expressions, numerous corpora of dynamic facial stimuli have been developed over the past two decades. The present research aimed to test existing sets of dynamic facial expressions (published between 2000 and 2015) in a cross-corpus validation effort. For this, 14 dynamic databases were selected that featured facial expressions of the basic six emotions (anger, disgust, fear, happiness, sadness, surprise) in posed or spontaneous form. In Study 1, a subset of stimuli from each database (N = 162) were presented to human observers and machine analysis, yielding considerable variance in emotion recognition performance across the databases. Classification accuracy further varied with perceived intensity and naturalness of the displays, with posed expressions being judged more accurately and as intense, but less natural compared to spontaneous ones. Study 2 aimed for a full validation of the 14 databases by subjecting the entire stimulus set (N = 3812) to machine analysis. A FACS-based Action Unit (AU) analysis revealed that facial AU configurations were more prototypical in posed than spontaneous expressions. The prototypicality of an expression in turn predicted emotion classification accuracy, with higher performance observed for more prototypical facial behavior. Furthermore, technical features of each database (i.e., duration, face box size, head rotation, and motion) had a significant impact on recognition accuracy. Together, the findings suggest that existing databases vary in their ability to signal specific emotions, thereby facing a trade-off between realism and ecological validity on the one end, and expression uniformity and comparability on the other.

show abstract

“…This process works well for larger samples; thus, spontaneous facial expressions were ideal for our study of 160 participants. Spontaneous expressions can also provide a benchmark for comparisons between different algorithms (Küster et al, 2020). We relied on a minimum of 90% of accurate facial analysis through all FaceReader analyses detected for each participant: each participant has been exposed to the website for a total of 3 min, and his/her facial expressions have been recorded for a total of 180 s. Only recordings that FaceReader processed properly for at least 162 s were considered.…”

Section: Methodsmentioning

confidence: 99%

Applying Implicit Association Test Techniques and Facial Expression Analyses in the Comparative Evaluation of Website User Experience

et al. 2021

View full text Add to dashboard Cite

This research project has the goal to verify whether the application of neuromarketing techniques, such as implicit association test (IAT) techniques and emotional facial expressions analyses may contribute to the assessment of user experience (UX) during and after website navigation. These techniques have been widely and positively applied in assessing customer experience (CX); however, little is known about their simultaneous application in the field of UX. As a specific context, the experience raised by different websites from two well-known automotive brands was compared. About 160 Italian university students were enrolled in an online experimental study. Participants performed a Brand Association Reaction Time Test (BARTT) version of the IAT where the two brands were compared according to different semantic dimensions already used in the automotive field. After completing the BARTT test, the participants navigated the target website: 80 participants navigated the first brand website, while the other half navigated the second brand website (between-subject design). During the first 3 min of website navigation, emotional facial expressions were recorded. The participants were asked to freely navigate the website home page, look for a car model and its characteristics and price, use the customising tool, and in the end, look for assistance. After the website navigation, all the participants performed, a second time, the BARTT version of the IAT, where the two brands were compared again, this time to assess whether the website navigation may impact the Implicit Associations previously detected. A traditional evaluation of the two websites was carried on by means of the classic heuristic evaluation. Findings from this study show, first of all, the significant results provided by neuromarketing techniques in the field of UX, as IAT can provide a positive application for assessing UX played by brand websites, thanks to the comparison of eventual changes in time reaction between the test performed before and after website navigation exposure. Secondly, results from emotional facial expression analyses during the navigation of both brand websites showed significant differences between the two brands, allowing the researchers to predict the emotional impact raised by each website. Finally, the positive correlation with heuristic evaluation shows that neuromarketing can be successfully applied in UX.

show abstract

Opportunities and Challenges for Using Automatic Human Affect Analysis in Consumer Research

Cited by 16 publications

References 67 publications

A performance comparison of eight commercially available automatic classifiers for facial affect recognition

A performance comparison of eight commercially available automatic classifiers for facial affect recognition

Human and machine validation of 14 databases of dynamic facial expressions

Applying Implicit Association Test Techniques and Facial Expression Analyses in the Comparative Evaluation of Website User Experience

Contact Info

Product

Resources

About