Generating Fillers Based on Dialog Act Pairs for Smooth Turn-Taking by Humanoid Robot

We address prediction of turn-taking considering related behaviors such as backchannels and fillers. Backchannels are used by the listeners to acknowledge that the current speaker can hold the turn. On the other hand, fillers are used by the prospective speakers to indicate a will to take a turn. We propose a turntaking model based on multitask learning in conjunction with prediction of backchannels and fillers. The multitask learning of LSTM neural networks shared by these tasks allows for efficient and generalized learning, and thus improves prediction accuracy. Evaluations with two kinds of dialogue corpora of human-robot interaction demonstrate that the proposed multitask learning scheme outperforms the conventional single-task learning.

show abstract

“…Several studies investigated the prosodic features of fillers [26,27], but there are only a limited studies on prediction of fillers [28,29].…”

Section: Filler Predictionmentioning

confidence: 99%

Prediction of Turn-taking Using Multitask Learning with Prediction of Backchannels and Fillers

et al. 2018

Self Cite

View full text Add to dashboard Cite

show abstract

“…Over the years, numerous scholars have extensively researched the prediction of pause fillers using various methods, which has resulted in more natural and authentic text generation. For instance, Nakanishi R. et al proposed a method based on analyzing humanrobot interaction data and machine learning models to predict the occurrence and appropriate forms of pause fillers, aiming to generate them at the beginning of system utterances in humanoid robot spoken dialog systems, indicating turn-taking or turn-holding intentions [1]. Balagopalan A. et al compared two common methods for AD detection on a matched dataset, assessing the advantages of domain knowledge and BERT pre-trained transfer models in predicting pauses and interruptions [2].…”

Section: Prediction Of Pause Fillersmentioning

confidence: 99%

An Improved Chinese Pause Fillers Prediction Module Based on RoBERTa

Yu,

Zhou,

Niu

2023

Applied Sciences

View full text Add to dashboard Cite

The prediction of pause fillers plays a crucial role in enhancing the naturalness of synthesized speech. In recent years, neural networks, including LSTM, BERT, and XLNet, have been employed for pause fillers prediction modules. However, these methods have exhibited relatively lower accuracy in predicting pause fillers. This paper introduces the utilization of the RoBERTa model for predicting Chinese pause fillers and presents a novel approach to training the RoBERTa model, effectively enhancing the accuracy of Chinese pause fillers prediction. Our proposed approach involves categorizing text from different speakers into four distinct style groups based on the frequency and position of Chinese pause fillers. The RoBERTa model is trained on these four groups of data, which incorporate different styles of fillers, thereby ensuring a more natural synthesis of speech. The Chinese pause fillers prediction module is evaluated on systems such as Parallel Tacotron2, FastPitch, and Deep Voice3, achieving a notable 26.7% improvement in word-level prediction accuracy compared to the BERT model, along with a 14% enhancement in position-level prediction accuracy. This substantial improvement results in a significant enhancement of the naturalness of the generated speech.

show abstract

“…2) Unlike the detection task, where audio alone can achieve reasonable performance (although lexical cues often help), the synthesis task requires suitable generation of both acoustic and lexical behaviors. When synthesizing fillers and backchannels, the meanings largely depend on their morphological forms [15,21,22,32]. To address this challenge, the majority of the synthesis task is still performed following a rule-based method.…”

Section: Introductionmentioning

confidence: 99%

I Know Your Feelings Before You Do: Predicting Future Affective Reactions in Human-Computer Dialogue

Inoue

Tian

et al. 2023

Extended Abstracts of the 2023 CHI Conference on Human Factors in Computing Systems

Self Cite

View full text Add to dashboard Cite

Current Spoken Dialogue Systems (SDSs) often serve as passive listeners that respond only after receiving user speech. To achieve human-like dialogue, we propose a novel future prediction architecture that allows an SDS to anticipate future affective reactions based on its current behaviors before the user speaks. In this work, we investigate two scenarios: speech and laughter. In speech, we propose to predict the user's future emotion based on its temporal relationship with the system's current emotion and its causal relationship with the system's current Dialogue Act (DA). In laughter, we propose to predict the occurrence and type of the user's laughter using the system's laughter behaviors in the current turn. Preliminary analysis of human-robot dialogue demonstrated synchronicity in the emotions and laughter displayed by the human and robot, as well as DA-emotion causality in their dialogue. This verifies that our architecture can contribute to the development of an anticipatory SDS. CCS CONCEPTS• Human-centered computing → Human computer interaction (HCI).

show abstract

Generating Fillers Based on Dialog Act Pairs for Smooth Turn-Taking by Humanoid Robot

Cited by 8 publications

References 14 publications

Prediction of Turn-taking Using Multitask Learning with Prediction of Backchannels and Fillers

Prediction of Turn-taking Using Multitask Learning with Prediction of Backchannels and Fillers

An Improved Chinese Pause Fillers Prediction Module Based on RoBERTa

I Know Your Feelings Before You Do: Predicting Future Affective Reactions in Human-Computer Dialogue

Contact Info

Product

Resources

About