Socially Interactive Robotic Platforms as Sign Language Tutors

Abstract

This paper investigates the role of interaction and communication kinesics in human–robot interaction. This study is part of a novel research project on sign language (SL) tutoring through interaction games with humanoid robots. The main goal is to motivate the children with communication problems to understand and imitate the signs implemented by the robot using basic upper torso gestures and sound. We present an empirical and exploratory study investigating the effect of basic nonverbal gestures consisting of hand movements, body and face gestures expressed by a humanoid robot, and having comprehended the word, the participants will give relevant feedback in SL. This way the participant is both a passive observer and an active imitator throughout the learning process in different phases of the game. A five-fingered R3 robot platform and a three-fingered Nao H-25 robot are employed within the games. Vision-, sound-, touch- and motion-based cues are used for multimodal communication between the robot, child and therapist/parent within the study. This paper presents the preliminary results of the proposed game tested with adult participants. The aim is to evaluate the SL learning ability of participants from a robot, and compare different robot platforms within this setup.

Robostar: An interaction game with humanoid robots for learning sign language

“…Yet, their average error rates are smaller in the games played with R3 than Nao in the first games (average error rate with R3=2.8; Nao=4.7), and total error in both games (R3=3.1; Nao=4.3). This agrees with the children's subjective evaluations of the games, such as "They find Nao cute, as a toy, and R3 is more like a peer or tutor, whose actions are more understandable" [21].…”

“…These children had same average error rate in both robots. Yet their error rate is significantly lower with R3, when they played with physically embodied robot platforms, instead of the virtual robots [21].…”

SPELTA: An expert system to generate therapy plans for speech and language disorders

“…More recently, Palmer et al (2012) argued that the advances in artificial intelligence and audiovisual signal processing make it possible to automate certain SLT tasks and thus deliver more intensive and efficient therapy out of the clinics, while making better use of the therapists' time. The state-of-the-art in this area includes web-based developments (Ooi, Raja, Sung, Fung, & Koh, 2012), mobile applications (Bunnell & Beidel, 2013) and various types of robots (Choe, Jung, Baird, & Grupen, 2013;Kose, Akalin, & Uluer, 2014), featuring components of voice-based and gesture-based interaction (Hogrefe, Ziegler, Wiesmayer, Weidinger, & Goldenberg, 2013;Sekine & Rose, 2013), avatars in the role of 'virtual therapists' (Abad et al, 2013;Teodoro, Martin, Keshner, Shi, & Rudnicky, 2013), etc. Expert systems have been used in this area, for example, to adapt parameters of the user interfaces, in order to make them more amenable and engaging (Kostoulas et al, 2012).…”

RAMSES : a robotic assistant and a mobile support environment for speech and language therapy

“…The results reveal very similar outcomes in the children responses for both cases (amount of words, filler words and key words). Other approaches propose the use of robots to support therapy sessions and improve the social interaction skill in children with Autism Spectrum Disorders [7], [8], motivate children with communication disorders to learn the sign language [9] or provide support for Mexican patients suffering from motor speech disorders (dysarthria) [10].…”