A Toolkit for the Automatic Analysis of Human Behavior in HCI Applications in the Wild

Abstract

Nowadays, smartphones and laptops equipped with cameras have become an integral part of our daily lives. The pervasive use of cameras enables the collection of an enormous amount of data, which can be easily extracted through video images processing. This opens up the possibility of using technologies that until now had been restricted to laboratories, such as eye-tracking and emotion analysis systems, to analyze users' behavior in the wild, during the interaction with websites. In this context, this paper introduces a toolkit that takes advantage of deep learning algorithms to monitor user's behavior and emotions, through the acquisition of facial expression and eye gaze from the video captured by the webcam of the device used to navigate the web, in compliance with the EU General data protection regulation (GDPR). Collected data are potentially useful to support user experience assessment of web-based applications in the wild and to improve the effectiveness of e-commerce recommendation systems.

Emotional Valence from Facial Expression as an Experience Audit Tool: An Empirical Study in the Context of Opera Performance

“…The emotion recognition system adopted in this study represents a state-of-the-art system for emotion recognition based on the analysis of facial expressions. Such software implements a combination of facial recognition and gaze-tracking technologies based on artificial intelligence algorithms, as described in detail in [45] and [49]. It enables the monitoring of the emotional state of people shot by a camera and their corresponding level of interest and involvement.…”

Smart retrofitting for human factors: a face recognition-based system proposal

“…This is made possible by the layered structure of a CNN, which adopts a more or less large number of hidden layers, with the aim of approaching the problem of identifying a face through an analysis carried out at the level of the single pixel. Some examples of CNNs applied to FR, but also to eye-gaze tracking, can be found in [14][15][16][17]. In these cases, the same and other information useful for the purpose of this paper are extracted (gender, age estimation, eye-gaze tracking, and FR), although they are applied to User Experience (UX) analysis.…”

How to Create Engaging Experiences From Human Emotions

“…The literature proposes several methods based on image processing and Convolution Neural Networks (CNN) that allows determining user gender and age, such as proposed in Ceccacci et al (2018). Some studies tested methods, based on regression and CNN, to track a person eye-gaze by using standard webcams or camera phones (Generosi et al, 2020;Krafka et al, 2016;Papoutsaki et al, 2016).…”

“…Infrared cameras were positioned in the Sferisterio arena to detect the facial audience expression during the show. The soft-ware adopted in this context implements a combination of facial recognition and gaze tracking technologies based on artificial intelligence algorithms, as described in detail in Generosi et al (2020) and Talipu et al (2019). It enables the age and gender recognition of people shot by the cameras, monitoring their emotional state and corresponding level of interest and involvement.…”