Enhancing Backchannel Prediction Using Word Embeddings

Ruede, Robin; Müller, Markus; Stüker, Sebastian; Waibel, Alex

doi:10.21437/interspeech.2017-1606

Cited by 20 publications

(38 citation statements)

References 10 publications

(13 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…It included 246 Cantonese conversations between trained assessors and older adult participants (171 males and 75 females), each being approximately 30 minutes in duration, and with hand transcriptions aligned at the word-level with speech. The scale of this dataset was comparable with some of the largest datasets used for backchanneling studies, such as SwDA [65,77,78]. Participants in this MoCA dataset were aged between 77 and 94, with an average age of 82.9.…”

Section: Discussionmentioning

confidence: 99%

“…Previous studies formulated the problem in different ways, e.g., focusing on prediction and choosing the same backchannels [34]; bundling multiple binary classifiers to predict different types of backchannels and giving corresponding actions [35]; training a multi-class classifier to predict and act at the same time [35]. Various machine learning algorithms have been applied, such as locally weighted linear regression [87], Hidden Markov Model (HMM) [58,59], Support Vector Machines [52], Long Short-Term Memory networks [28,34,77,78] and hybrid time-delay neural network (TDNN)/HMM system [65]. Feature engineering is another important component for MLbased methods.…”

Section: Model-based Backchanneling Using Machine Learning Algorithmsmentioning

confidence: 99%

See 1 more Smart Citation

TalkTive: A Conversational Agent Using Backchannels to Engage Older Adults in Neurocognitive Disorders Screening

Ding¹,

Kang²,

HO³

et al. 2022

Preprint

View full text Add to dashboard Cite

Conversational agents (CAs) have the great potential in mitigating the clinicians' burden in screening for neurocognitive disorders among older adults. It is important, therefore, to develop CAs that can be engaging, to elicit conversational speech input from older adult participants for supporting assessment of cognitive abilities. As an initial step, this paper presents research in developing the backchanneling ability in CAs in the form of a verbal response to engage the speaker. We analyzed 246 conversations of cognitive assessments between older adults and human assessors, and derived the categories of reactive backchannels (e.g. "hmm") and proactive backchannels (e.g. "please keep going"). This is used in the development of TalkTive, a CA which can predict both timing and form of backchanneling during cognitive assessments. The study then invited 36 older adult participants to evaluate the backchanneling feature. Results show that proactive backchanneling is more appreciated by participants than reactive backchanneling. CCS CONCEPTS• Human-centered computing → Empirical studies in HCI; Auditory feedback; Empirical studies in interaction design; Soundbased input / output.

show abstract

Section: Discussionmentioning

confidence: 99%

Section: Model-based Backchanneling Using Machine Learning Algorithmsmentioning

confidence: 99%

TalkTive: A Conversational Agent Using Backchannels to Engage Older Adults in Neurocognitive Disorders Screening

Ding¹,

Kang²,

HO³

et al. 2022

Preprint

View full text Add to dashboard Cite

show abstract

“…We now refined this approach by evaluating different methods to add timed word embeddings via word2vec. Comparing the performance using various feature combinations, we observed that adding linguistic features improved the performance over a prediction system that only uses acoustic features [8]. The most commonly used acoustic features in related research are fast and slow voice pitch slopes and pauses of varying lengths.…”

Section: Non-verbal Cues As Feedback For the Usermentioning

confidence: 91%

Speech interaction strategies for a humanoid assistant

et al. 2018

Self Cite

View full text Add to dashboard Cite

Abstract. The goal of SecondHands, a H2020 project, is to design a robot that can offer help to a maintenance technician in a proactive manner. The robot is to act as a second pair of hands that can assist the technician when he is in need of help. In order for the robot to be of real help to the technician, it needs to understand his needs and follow his commands. Interaction via speech is a crucial part of this. Due to the nature of the situation in which the interactions take place, often the technician needs to speak to the robot when under stress performing strenuous physical labor, the classical turn based interaction schemes need to be transformed into dialogue systems that perform stream processing, anticipating user intentions, correcting itself as more information become available, in order to be able to respond in a rapid manner. In order to meet these demands, we are developing low-latency streaming based automatic speech recognition systems in combination with recurrent neural network based Natural Language Understanding systems that perform slot filling and intent recognition in order for the robot to provide assistance in a rapid manner, that can be partly based on speculative classifications that are then being refined as more speech becomes available.

show abstract

“…Prior work predicting initiation points uses prosodic features like pitch and frequency variation with bag-of-embeddings to predict backchannels (Ruede et al, 2017a) and turn-completion (Skantze, 2017), and more recently, Ekstedt and Skantze Figure 1: Humans produce overlapping speech with small gaps. By predicting lead to initiation, virtual agents can respond without long waiting periods (2021) finetuned GPT-2 on dialogue datasets to predict turn-completion using only word features.…”

Section: Introductionmentioning

confidence: 99%

When can I Speak? Predicting initiation points for spoken dialogue agents

Li¹,

Paranjape²,

Manning³

2022

Preprint

View full text Add to dashboard Cite

Current spoken dialogue systems initiate their turns after a long period of silence (700-1000ms), which leads to little real-time feedback, sluggish responses, and an overall stilted conversational flow. Humans typically respond within 200ms and successfully predicting initiation points in advance would allow spoken dialogue agents to do the same. In this work, we predict the lead-time to initiation using prosodic features from a pre-trained speech representation model (wav2vec 1.0) operating on user audio and word features from a pre-trained language model (GPT-2) operating on incremental transcriptions. To evaluate errors, we propose two metrics w.r.t. predicted and true lead times. We train and evaluate the models on the Switchboard Corpus and find that our method outperforms features from prior work on both metrics and vastly outperforms the common approach of waiting for 700ms of silence.

show abstract

Enhancing Backchannel Prediction Using Word Embeddings

Cited by 20 publications

References 10 publications

TalkTive: A Conversational Agent Using Backchannels to Engage Older Adults in Neurocognitive Disorders Screening

TalkTive: A Conversational Agent Using Backchannels to Engage Older Adults in Neurocognitive Disorders Screening

Speech interaction strategies for a humanoid assistant

When can I Speak? Predicting initiation points for spoken dialogue agents

Contact Info

Product

Resources

About