Utilizing an Emotional Robot Capable of Lip-Syncing in Robot-Assisted Speech Therapy Sessions for Children with Language Disorders

Abstract

This study scrutinizes the impacts of utilizing a socially assistive robot, the RASA robot, during speech therapy sessions for children with language disorders. Two capabilities were developed for the robotic platform to enhance children-robot interactions during speech therapy interventions: facial expression communication (containing recognition and expression) and lip-syncing. Facial expression recognition was conducted by training several well-known CNN architectures on one of the most extensive facial expressions databases, the AffectNet database, and then modifying them using the transfer learning strategy performed on the CK+ dataset. The robot’s lip-syncing capability was designed in two steps. The first step was concerned with designing precise schemes of the articulatory elements needed during the pronunciation of the Persian phonemes (i.e., consonants and vowels). The second step included developing an algorithm to pronounce words by disassembling them into their components (including consonants and vowels) and then morphing them into each other successively. To pursue the study’s primary goal, two comparable groups of children with language disorders were considered, the intervention and control groups. The intervention group attended therapy sessions in which the robot acted as the therapist’s assistant, while the control group only communicated with the human therapist. The study’s first purpose was to compare the children’s engagement while playing a mimic game with the affective robot and the therapist, conducted via video coding. The second objective was to assess the efficacy of the robot’s presence in the speech therapy sessions alongside the therapist, accomplished by administering the Persian Test of Language Development, Persian TOLD. According to the first scenario, playing with the affective robot is more engaging than playing with the therapist. Furthermore, the statistical analysis of the study’s results indicates that participating in robot-assisted speech therapy (RAST) sessions enhances children with language disorders’ achievements in comparison with taking part in conventional speech therapy interventions.

A feasibility study of using Kaspar, a humanoid robot for speech and language therapy for children with learning disabilities*

“…Their findings indicated that NAO had potential to increase motivation, and readiness towards learning as well as to improve attention span of children. Another pilot study, using the RASA robot in speech therapy sessions for children with language disorders found further supporting evidence for the potential usability of robots in SLC therapy with children [16]. These studies, however, both presented preliminary exploratory findings with small number of participants.…”

Robot-Assisted Language Education and Speech Therapy for Children with Cleft Lip and Palate

“…Indeed, gamification has the potential to enhance student motivation and increase engagement (Zourmpakis et al, 2022(Zourmpakis et al, , 2023Papadakis et al, 2022, particularly when combined with social robots (Yang et al, 2023;Tran et al, 2023;Issa et al, 2023). Robots provided many enticements to children, which urge them to continue therapy (Alwadain, Al-Ma'aitah, & Saad, 2020;Spaulding et al, 2018), and interaction with robots has been shown to help children with Autism Spectrum Disorder (ASD) and Delayed Speech Development (DSD) improve their social and communication skills (Bouhali et al, 2023;Amirova et al, 2023;Esfandbod et al, 2023); furthermore, in studies carried out by Warren et al (2015), children with speech disorders responded better to humanoid robots than adult administrators or typically developing counterparts.…”

A Systematic Literature Review on the Applications of Robots and Natural Language Processing in Education

“…14 Esfandbd et al [74] Examine the effects of using RASA robot in speech therapy for children with language disorders.…”