Part-of-Speech and Prosody-based Approaches for Robot Speech and Gesture Synchronization

Abstract

Humanoid robots are already among us and they are beginning to assume more social and personal roles, like guiding and assisting people. Thus, they should interact in a human-friendly manner, using not only verbal cues but also synchronized non-verbal and para-verbal cues. However, available robots are not able to communicate in this multimodal way, being just able to perform predefined gesture sequences, handcrafted to accompany specific utterances. In the current paper, we propose a model based on three different approaches to extend humanoid robots communication behaviour with upper body gestures synchronized with the speech for novel utterances, exploiting part-of-speech grammatical information, prosody cues, and a combination of both. User studies confirm that our methods are able to produce natural, appropriate and good timed gesture sequences synchronized with speech, using both beat and emblematic gestures.

A Social Robot Assisting in Cognitive Stimulation Therapy

“…Speech and gestures are synchronized by dividing the speech in smaller audio chunks, and then generating motions for each chunk, with the same duration. That same year, Pérez-Mayos et al [196] proposed a model that uses three different approaches for speech-gesture synchronization. The first approach starts by identifying keywords in the text connected to gestures in the database.…”

“…Another example is the work of Kucherenko et al [112], where the BERT encoding of the speech transcription and temporal information about how the sentence is uttered is combined with log-power mel-spectogram features extracted from the audio signal. Pérez-mayos et al [196] decided to use the prosody of the speech to select beat gestures, while text is used for the remaining categories.…”

“…Their model then predicts which one of the clusters obtained should be selected based on the speech, and a gesture is instantiated based on the position of said cluster's centroid. Authors like Pérez-Mayos et al [196] or Xiao et al [203] handcrafted their gestures using the tools provided by the robotic platforms used in their works (in both cases, a Nao robot). Pérez-Mayos's work synchronized these predefined expressions by associating their triggering to either keywords or pitch peak patterns in the utterance.…”

“…Ishi et al [192] decided to use conditional probabilities to represent the connections between speech content and gesture functions, and between functions and motions. Pérez-Mayos et al [196] combined a part-of-speech based approach, where gestures were associated with keywords in the speech, while beat gestures were generated based on the prosodic content (in particular, the pitch graph). Finally, the work presented by Xiao et al [203] focuses on finding the correlation between the dissimilarities between two sentences and the dissimilarities between their associated behaviours.…”

Modeling and evaluating beat gestures for social robots

“…Their joints have different degrees of freedom (DOF), movable ranges are not the same, etc. Therefore, original motions must be modified to be feasible by the robot, i.e the captured movements must be correctly mapped by satisfying several constraints (see [22] for a good overview of every aspect of the motion imitation task).…”