A Multimodal Approach for Real Time Recognition of Engagement towards Adaptive Serious Games for Health

Abstract

In this article, an unobtrusive and affordable sensor-based multimodal approach for real time recognition of engagement in serious games (SGs) for health is presented. This approach aims to achieve individualization in SGs that promote self-health management. The feasibility of the proposed approach was investigated by designing and implementing an experimental process focusing on real time recognition of engagement. Twenty-six participants were recruited and engaged in sessions with a SG that promotes food and nutrition literacy. Data were collected during play from a heart rate sensor, a smart chair, and in-game metrics. Perceived engagement, as an approximation to the ground truth, was annotated continuously by participants. An additional group of six participants were recruited for smart chair calibration purposes. The analysis was conducted in two directions, firstly investigating associations between identified sitting postures and perceived engagement, and secondly evaluating the predictive capacity of features extracted from the multitude of sources towards the ground truth. The results demonstrate significant associations and predictive capacity from all investigated sources, with a multimodal feature combination displaying superiority over unimodal features. These results advocate for the feasibility of real time recognition of engagement in adaptive serious games for health by using the presented approach.

Artificial Intelligence–Driven Serious Games in Health Care: Scoping Review

“…This progression comes naturally with the increasing adoption of connected devices, including wearable and nonwearable sensors, as part of the AI-driven serious game. With this in mind, we project that AI-driven serious games will be more adaptable in an unobtrusive and affordable manner [94]. This review found that 3D serious games were slightly more common than 2D serious games, which is in line with the findings of a previous review [36].…”

“…This progression comes naturally with the increasing adoption of connected devices, including wearable and nonwearable sensors, as part of the AI-driven serious game. With this in mind, we project that AI-driven serious games will be more adaptable in an unobtrusive and affordable manner [ 94 ].…”

The ENDORSE Feasibility Study: Exploring the Use of M-Health, Artificial Intelligence and Serious Games for the Management of Childhood Obesity

“…The low engagement with the ENDORSE’s SG highlights the great challenge of designing an SG featuring a balance between user attractiveness and scientific background in accordance with the existing literature [ 53 , 54 , 55 ]. However, the significant enhancement that was observed in terms of engagement in the pilot version compared to the pre-pilot version supports the effectiveness of the SG’s gameplay flow and conceptual framework.…”

EEG-Based Neural Synchrony Predicts Evaluative Engagement with Music Videos

“…For FAA, this was shown to elicit weaker activity patterns than multimodal stimuli that most of the previous studies employed [25]. Neural synchronization may also be diminished due to this unimodal approach, since engagement in general was higher with multimodal design [26].…”