Facial Electromyography-based Adaptive Virtual Reality Gaming for Cognitive Training

Abstract

Cognitive training has shown promising results for delivering improvements in human cognition related to attention, problem solving, reading comprehension and information retrieval. However, two frequently cited problems in cognitive training literature are a lack of user engagement with the training programme, and a failure of developed skills to generalise to daily life. This paper introduces a new cognitive training (CT) paradigm designed to address these two limitations by combining the benefits of gamification, virtual reality (VR), and affective adaptation in the development of an engaging, ecologically valid, CT task. Additionally, it incorporates facial electromyography (EMG) as a means of determining user affect while engaged in the CT task. This information is then utilised to dynamically adjust the game's difficulty in real-time as users play, with the aim of leading them into a state of flow. Affect recognition rates of 64.1% and 76.2%, for valence and arousal respectively, were achieved by classifying a DWT-Haar approximation of the input signal using kNN. The affect-aware VR cognitive training intervention was then evaluated with a control group of older adults. The results obtained substantiate the notion that adaptation techniques can lead to greater feelings of competence and a more appropriate challenge of the user's skills. CCS CONCEPTS • Human-centered computing → Interactive systems and tools; • Computing methodologies → Virtual reality; Machine learning.

Electromyography Acquisition and Intervention in Healthcare for Older Adults: A HCI Scoping Review

“…-Smart wheelchairs [13] -Robot for daily hand tasks and rehabilitation [14], [26] -Toolbox for the muscle data analysis [15] Gamified therapy 11 -Virtual reality training and Electromyographic biofeedback [43] -VR or AR motor imagery [44] -Serious game [29] Wearable 12 -Electromyographic sensor placement [45] -Electrical stimulation patches [28], [42] Interface design 14 -Facial expression classified [21] -Hand gesture controlling [37] -Brain computer interface [46] Due to the necessity of wearing electromyographic sensors and electrical stimulation devices, a total of 7, 6, and 4 articles respectively discussed the interdisciplinary applications of wearable technology in Assistive Technology and Tool Design, Gamified Therapy, and Interface design. Furthermore, among the papers reviewed, four of them shared a common theme of Assistive Technology and Tool Design, along with Interface Design.…”

Visualization and Interaction Technologies in Serious and Exergames for Cognitive Assessment and Training: A Survey on Available Solutions and Their Validation

“…Immersive HMDs [6], [14], [17], [20], [27], [29], [35], [42], [46], [47], [52], [174] Touchful devices comprise the widely diffused tools for gaming (N=181) and physiotherapic devices (N=17), e.g. treadmills, cycle-ergometers and pressure sensitive plates, which are particularly suitable for rehabilitation programs aimed at improving motor skills in subjects with age-related disorders [15], [50], [88], [94], [111], [146]- [149], [203], [208], major neurocognitive disorders (MNCD) [23], stroke [115], [151] and multiple sclerosis (MS) [22] patients.…”

“…As participants could suffer from visual and hearing problems or have troubles with atten- tion, visual scenes have to be as simple as possible and search space should be reduced. During gameplay, simple and immediate data as a countdown timer, the number of remained chances to guess an answer, the current score or achieved goals are often shown to incentive the subject to answer as quickly and accurately as possible [3], [14], [17], [19], [20], [22], [24], [26], [28]- [31], [33], [38]- [41], [44], [49], [50], [52], [53], [71], [72], [79], [83], [96], [97], [99], [100], [106], [109], [115], [120], [124], [125], [130]- [132], [137], [138], [142], [150], [161], [164], [179], [191], [209], [211], [212], [231],…”

“…Controller, joystick or gamepad [6], [14], [15], [17], [20], [35], [42], [50], [52], [54]- [56], [62], [65], [70], [79],…”

A Comparative Study of Colour Effects on Cognitive Performance in Real-World and VR Environments

“…In numerous fields, the benefits of VR emerge from its power to build identifiable immersive experiences that motivate research and applications, including building planning, medical training, entertainment, design decision-making, data visualisation, and marketing [ 12 , 13 , 14 , 15 , 16 , 17 , 18 ]. Moreover, a series of recent clinical research that utilised immersive VR technology to deliver aversion therapy (i.e., specific phobias, social anxiety, post-traumatic stress disorder, depression, eating disorders, and paranoia assessment) has revealed that the results of VR treatments are equal to or more positive than those of traditional treatment methods [ 19 , 20 , 21 , 22 , 23 , 24 , 25 , 26 , 27 , 28 , 29 , 30 ].…”