Automatic assessment of cognitive and emotional states in virtual reality‐based flexibility training for four adolescents with autism

Abstract

Tracking students’ learning states to provide tailored learner support is a critical element of an adaptive learning system. This study explores how an automatic assessment is capable of tracking learners’ cognitive and emotional states during virtual reality (VR)‐based representational‐flexibility training. This VR‐based training program aims to promote the flexibility of adolescents with autism spectrum disorder (ASD) in interpreting, selecting and creating multimodal representations during STEM‐related design problem solving. For the automatic assessment, we used both natural language processing (NLP) and machine‐learning techniques to develop a multi‐label classification model. We then trained the model with the data from a total of audio‐ and video‐recorded 66 training sessions of four adolescents with ASD. To validate the model, we implemented both k‐fold cross‐validations and the manual evaluations by expert reviewers. The study finding suggests the feasibility of implementing the NLP and machine‐learning driven automatic assessment to track and assess the cognitive and emotional states of individuals with ASD during VR‐based flexibility training. The study finding also denotes the importance and viability of providing adaptive supports to maintain learners’ cognitive and affective engagement in a highly interactive digital learning environment.

Virtual Reality Technology as an Educational and Intervention Tool for Autism Spectrum Disorder: Current Perspectives and Future Directions

“…For example, researchers and engineers on VR technology development can investigate how to combine VR technologies with AI and machine-learning techniques to promote customized intervention and provide tailored support. Automatic recognition and assessment technology including automatic speech recognition and evaluation [176,177], automatic assessment of cognitive and emotional states [178] need to be embedded into the VR training systems to measure and track users' abilities and states and deliver immediate and timely feedback prior to, during, and after training. The incorporation of accurate, intelligent, and efficient automatic evaluation techniques can greatly ease the workload of speech-language and cognitive-behavioral therapists and more importantly, offer much-needed assessment and intervention services for the ASD population living in less developed regions that lack trained and qualified therapists.…”

Applications of virtual reality (VR) in autism research: current trends and taxonomy of definitions

“…Cognitive training in VR research has also become the focus of attention over the last few years (Didehbani et al. , 2016; Moon et al. , 2020), along with the design and adoption of education-based VR technologies.…”

“…For example, Simões et al (2020) and Miller et al (2020) investigated the effectiveness of virtual travel training for autism and highlighted several promising avenues for potentially fostering independence through future technology-based programmes. Cognitive training in VR research has also become the focus of attention over the last few years (Didehbani et al, 2016;Moon et al, 2020), along with the design and adoption of education-based VR technologies. For example, investigated the usability and learner experience of VR adaptive skills intervention for autistic adults.…”

To social with social distance: a case study on a VR-enabled graduation celebration amidst the pandemic

“…Carefully designed scenarios in VR also help people with special needs. For instance, researchers conducted training sessions in VR to help users with autism or schizophrenia improve their social skills (e.g., Cox et al 2017 ; Moon et al 2020 ; Smith et al 2015 ; Wang et al 2018 ).…”

“…One of the salient places for MUVE research and practices is at the education sector, in which learning content can be simulated, real-life contexts can be created and repeated, physical and social interactions can be achieved, and details can be deliberately included (Garris et al 2002 ; Teoh 2012 ). In the educational settings, 3D desktop MUVEs such as SecondLife and OpenSimulator are used to teach content knowledge (e.g., Chen 2016 ; Halvorson et al 2011 ; Wang and Braman 2009 ; Zhang 2013 ), train pre-service teachers or student instructors (e.g., Gregory and Masters 2012 ; Ke and Xu 2020 ; Muir et al 2013 ), help students with special needs (e.g., Ke et al 2015 ; Wang et al 2018 ; Moon et al 2020 ), and so on. One of the foundational theories underneath these studies is the constructionists’ point of view which believes that the effective understanding of the world happens when people actively engage meaningful interactions with the environment around them (Papert and Harel 1991 ).…”