Evaluation of a Korean Lip-sync system for an android robot

Hyung, Hyun-Jun; Ahn, Byeong-Kyu; Choi, Dongwoon; Lee, Duk-Yeon; Lee, Dong‐Wook

doi:10.1109/urai.2016.7734025

Cited by 5 publications

(2 citation statements)

References 9 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…A recent speech to video application for virtual characters by [30] implements the MFCC method of audio signal analysis and derived compelling results. The application has a 0.35s response time to generate lip synchronisation patterns for the virtual characters, which is higher than [7] model of 0.5s and [28] 0.4s-3.14s robotic mouth response time . Thus, the MFCC approach is significant in developing a robotic mouth that can respond to incoming live audio transmissions due to its high speed of computation and a low field of noise interference.…”

Section: The Uncanny Valleymentioning

confidence: 94%

“…This mouth configuration is significant, as discussed previously, immobile or muted robotic lip actuation has a high potential to be interpreted as aggressive or unemotional and generate negative perceptual stimulus. In support, [28] developed a robotic mouth system to examine common lip-syncing factors affecting humanoid robots. However, the study claims that the robot developed for this research can perform complex mouth shapes for replicating human vowel and consonant lip patterns.…”

Section: The Uncanny Valleymentioning

confidence: 99%

See 1 more Smart Citation

A Novel Speech to Mouth Articulation System for Realistic Humanoid Robots

Strathearn

2021

J Intell Robot Syst

View full text Add to dashboard Cite

A significant ongoing issue in realistic humanoid robotics (RHRs) is inaccurate speech to mouth synchronisation. Even the most advanced robotic systems cannot authentically emulate the natural movements of the human jaw, lips and tongue during verbal communication. These visual and functional irregularities have the potential to propagate the Uncanny Valley Effect (UVE) and reduce speech understanding in human-robot interaction (HRI). This paper outlines the development and testing of a novel Computer Aided Design (CAD) robotic mouth prototype with buccinator actuators for emulating the fluidic movements of the human mouth. The robotic mouth system incorporates a custom Machine Learning (ML) application that measures the acoustic qualities of speech synthesis (SS) and translates this data into servomotor triangulation for triggering jaw, lip and tongue positions. The objective of this study is to improve current robotic mouth design and provide engineers with a framework for increasing the authenticity, accuracy and communication capabilities of RHRs for HRI. The primary contributions of this study are the engineering of a robotic mouth prototype and the programming of a speech processing application that achieved a 79.4% syllable accuracy, 86.7% lip synchronisation accuracy and 0.1s speech to mouth articulation differential.

show abstract

Section: The Uncanny Valleymentioning

confidence: 94%

Section: The Uncanny Valleymentioning

confidence: 99%

A Novel Speech to Mouth Articulation System for Realistic Humanoid Robots

Strathearn

2021

J Intell Robot Syst

View full text Add to dashboard Cite

show abstract

Utilizing an Emotional Robot Capable of Lip-Syncing in Robot-Assisted Speech Therapy Sessions for Children with Language Disorders

Esfandbod

Rokhi

Meghdari

et al. 2022

Int J of Soc Robotics

View full text Add to dashboard Cite

This study scrutinizes the impacts of utilizing a socially assistive robot, the RASA robot, during speech therapy sessions for children with language disorders. Two capabilities were developed for the robotic platform to enhance children-robot interactions during speech therapy interventions: facial expression communication (containing recognition and expression) and lip-syncing. Facial expression recognition was conducted by training several well-known CNN architectures on one of the most extensive facial expressions databases, the AffectNet database, and then modifying them using the transfer learning strategy performed on the CK+ dataset. The robot’s lip-syncing capability was designed in two steps. The first step was concerned with designing precise schemes of the articulatory elements needed during the pronunciation of the Persian phonemes (i.e., consonants and vowels). The second step included developing an algorithm to pronounce words by disassembling them into their components (including consonants and vowels) and then morphing them into each other successively. To pursue the study’s primary goal, two comparable groups of children with language disorders were considered, the intervention and control groups. The intervention group attended therapy sessions in which the robot acted as the therapist’s assistant, while the control group only communicated with the human therapist. The study’s first purpose was to compare the children’s engagement while playing a mimic game with the affective robot and the therapist, conducted via video coding. The second objective was to assess the efficacy of the robot’s presence in the speech therapy sessions alongside the therapist, accomplished by administering the Persian Test of Language Development, Persian TOLD. According to the first scenario, playing with the affective robot is more engaging than playing with the therapist. Furthermore, the statistical analysis of the study’s results indicates that participating in robot-assisted speech therapy (RAST) sessions enhances children with language disorders’ achievements in comparison with taking part in conventional speech therapy interventions.

show abstract