Identifying Features of Bodily Expression As Indicators of Emotional Experience during Multimedia Learning

Abstract

The importance of emotions experienced by learners during their interaction with multimedia learning systems, such as serious games, underscores the need to identify sources of information that allow the recognition of learners’ emotional experience without interrupting the learning process. Bodily expression is gaining in attention as one of these sources of information. However, to date, the question of how bodily expression can convey different emotions has largely been addressed in research relying on acted emotion displays. Following a more contextualized approach, the present study aims to identify features of bodily expression (i.e., posture and activity of the upper body and the head) that relate to genuine emotional experience during interaction with a serious game. In a multimethod approach, 70 undergraduates played a serious game relating to financial education while their bodily expression was captured using an off-the-shelf depth-image sensor (Microsoft Kinect). In addition, self-reports of experienced enjoyment, boredom, and frustration were collected repeatedly during gameplay, to address the dynamic changes in emotions occurring in educational tasks. Results showed that, firstly, the intensities of all emotions indeed changed significantly over the course of the game. Secondly, by using generalized estimating equations, distinct features of bodily expression could be identified as significant indicators for each emotion under investigation. A participant keeping their head more turned to the right was positively related to frustration being experienced, whereas keeping their head more turned to the left was positively related to enjoyment. Furthermore, having their upper body positioned more closely to the gaming screen was also positively related to frustration. Finally, increased activity of a participant’s head emerged as a significant indicator of boredom being experienced. These results confirm the value of bodily expression as an indicator of emotional experience in multimedia learning systems. Furthermore, the findings may guide developers of emotion recognition procedures by focusing on the identified features of bodily expression.

A survey on automatic engagement recognition methods: online and traditional classroom

“…Also, body of the participant positioned nearer to the gaming screen is related to frustration and as time passed leading to boredom [64]. Garger [65], the frequency of head tilting is associated with boredom, conforms in the findings of [63]. The conclusions of these studies help researchers to label their data.…”

“…Riemer et al [63] studied the effect of reactions undergone by students during the interaction session with a multimedia learning system. To identify the features of bodily expression, the body actions of seventy undergraduate students while playing the serious game related to financial education was captured.…”

Mental Model Development in Multimedia Learning: Interrelated Effects of Emotions and Self-Monitoring

“…Moreover, repeated self-reports of emotions may distort the effects of emotions on cognitive and metacognitive processes (Spering et al, 2005). Therefore, future studies should seek to apply objective and less invasive measures of emotions, for example, by using input devices (e.g., Miller and Mandryk, 2016) or recording facial or bodily expression features (e.g., Wiklund et al, 2015; Riemer et al, 2017).…”

Implementation of The Psychowriting Methods to Improve Linguistic Intelligence for Children in the Field of Writing Interest

“…Learning indicators are changes in behaviour. The learning process is said to be successful if it results in behavioural changes in individuals (Dlouhá, Barton, Janoušková, & Dlouhý, 2013;Riemer, Frommel, Layher, Neumann, & Schrader, 2017;Wu et al, 2020). Psychowriting is a method of an approach to learning that is associated with writing skills with psychology through different special treatments for each personality.…”