Semi-Supervised Behavior Labeling Using Multimodal Data during Virtual Teamwork-Based Collaborative Activities

Plunk, Abigale; Amat, Ashwaq Zaini; Tauseef, Mahrukh; Peters, Richard Alan; Sarkar, Nilanjan

doi:10.3390/s23073524

Cited by 3 publications

(1 citation statement)

References 33 publications

(32 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…On the other hand, the HMM prediction results for Waiting and Struggling were reliable since the temporal information that was learned from the training was embedded within the state transition and emission probability matrices. This is consistent with the results reported by another study that implemented a semi-supervised model using the same dataset as this study [67]. The study compared the performance of the developed semi-supervised automated labeling of behaviors to supervised and unsupervised models.…”

Section: Prediction Models Evaluation Resultssupporting

confidence: 90%

Prediction Models of Collaborative Behaviors in Dyadic Interactions: An Application for Inclusive Teamwork Training in Virtual Environments

Amat,

Plunk,

Adiani

et al. 2024

Signals

Self Cite

View full text Add to dashboard Cite

Collaborative virtual environment (CVE)-based teamwork training offers a promising avenue for inclusive teamwork training. The incorporation of a feedback mechanism within virtual training environments can enhance the training experience by scaffolding learning and promoting active collaboration. However, an effective feedback mechanism requires a robust prediction model of collaborative behaviors. This paper presents a novel approach using hidden Markov models (HMMs) to predict human behavior in collaborative interactions based on multimodal signals collected from a CVE-based teamwork training simulator. The HMM was trained using k-fold cross-validation, achieving an accuracy of 97.77%. The HMM was evaluated against expert-labeled data and compared against a rule-based prediction model, demonstrating the superior predictive capabilities of the HMM, with the HMM achieving 90.59% accuracy compared to 76.53% for the rule-based model. These results highlight the potential of HMMs to predict collaborative behaviors that could be used in a feedback mechanism to enhance teamwork training experiences despite the complexity of these behaviors. This research contributes to advancing inclusive and supportive virtual learning environments, bridging gaps in cross-neurotype collaborations.

show abstract

Section: Prediction Models Evaluation Resultssupporting

confidence: 90%

Prediction Models of Collaborative Behaviors in Dyadic Interactions: An Application for Inclusive Teamwork Training in Virtual Environments

Amat,

Plunk,

Adiani

et al. 2024

Signals

Self Cite

View full text Add to dashboard Cite

show abstract

Detection of Stress Stimuli in Learning Contexts of iVR Environments

Ramírez-Sanz,

Peña-Alonso,

Serrano-Mamolar

et al. 2023

Lecture Notes in Computer Science

View full text Add to dashboard Cite

Pilot study of a real-time early agitation capture technology (REACT) for children with intellectual and developmental disabilities

Khan,

Plunk,

Zheng

et al. 2024

DIGITAL HEALTH

View full text Add to dashboard Cite

Objective Children and adolescents with intellectual and developmental disabilities (IDD), particularly those with autism spectrum disorder, are at increased risk of challenging behaviors such as self-injury, aggression, elopement, and property destruction. To mitigate these challenges, it is crucial to focus on early signs of distress that may lead to these behaviors. These early signs might not be visible to the human eye but could be detected by predictive machine learning (ML) models that utilizes real-time sensing. Current behavioral assessment practices lack such proactive predictive models. This study developed and pilot-tested real-time early agitation capture technology (REACT), a real-time multimodal ML model to detect early signs of distress, termed “agitations.” Integrating multimodal sensing, ML, and human expertise could make behavioral assessments for people with IDD safer and more efficient. Methods We leveraged wearable technology to collect behavioral and physiological data from three children with IDD aged 6 to 9 years. The effectiveness of the REACT system was measured using F1 score, assessing its usefulness at the time of agitation to 20s prior. Results The REACT system was able to detect agitations with an average F1 score of 78.69% at the time of agitation and 68.20% 20s prior. Conclusion The findings support the use of the REACT model for real-time, proactive detection of agitations in children with IDD. This approach not only improves the accuracy of detecting distress signals that are imperceptible to the human eye but also increases the window for timely intervention before behavioral escalation, thereby enhancing safety, well-being, and inclusion for this vulnerable population. We believe that such technological support system will enhance user autonomy, self-advocacy, and self-determination.

show abstract

Semi-Supervised Behavior Labeling Using Multimodal Data during Virtual Teamwork-Based Collaborative Activities

Cited by 3 publications

References 33 publications

Prediction Models of Collaborative Behaviors in Dyadic Interactions: An Application for Inclusive Teamwork Training in Virtual Environments

Prediction Models of Collaborative Behaviors in Dyadic Interactions: An Application for Inclusive Teamwork Training in Virtual Environments

Detection of Stress Stimuli in Learning Contexts of iVR Environments

Pilot study of a real-time early agitation capture technology (REACT) for children with intellectual and developmental disabilities

Contact Info

Product

Resources

About