BPM_MT: Enhanced Backchannel Prediction Model using Multi-Task Learning

Jang, Jin Yea; Kim, San; Jung, Myung-Pyo; Shin, Saim; Gweon, Gahgene

doi:10.18653/v1/2021.emnlp-main.277

Cited by 8 publications

(16 citation statements)

References 21 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Similar to Ruede [9], both models predicted BC categories in 512 ms increments. As shown in Table 2, the proposed model outperformed the comparison BCP models in Ortega and others [8] and Jang and others [7] in terms of the macro F1-score by up to 18.7% and 17.9%, respectively.…”

Section: Resultsmentioning

confidence: 81%

“…BCP was the focus on Ortega and others [8] and Jang and others [7]. Nevertheless, training in Ruede [9], which focused on BOP, aligned with the procedures of these two studies.…”

Section: Resultsmentioning

confidence: 99%

“…The BCP model used a convolutional neural network to predict BC categories, and listener embeddings captured unique listener characteristics. In Jang and others [7], BCP was performed during counseling sessions involving physicians. Multitask learning was adopted to predict BC categories and classify emotions simultaneously.…”

Section: Related Workmentioning

confidence: 99%

“…Various BC prediction methods have been developed [6][7][8][9]. Ruede [9] introduced multilayer perceptrons (MLPs) and recurrent neural networks (RNNs) for BOP and used a 1.5 s context input to consider the data balance between instances with and without BCs.…”

Section: Introductionmentioning

confidence: 99%

See 3 more Smart Citations

Joint streaming model for backchannel prediction and automatic speech recognition

Choi,

Bang,

Kim

2024

ETRI Journal

View full text Add to dashboard Cite

In human conversations, listeners often utilize brief backchannels such as “uh‐huh” or “yeah.” Timely backchannels are crucial to understanding and increasing trust among conversational partners. In human–machine conversation systems, users can engage in natural conversations when a conversational agent generates backchannels like a human listener. We propose a method that simultaneously predicts backchannels and recognizes speech in real time. We use a streaming transformer and adopt multitask learning for concurrent backchannel prediction and speech recognition. The experimental results demonstrate the superior performance of our method compared with previous works while maintaining a similar single‐task speech recognition performance. Owing to the extremely imbalanced training data distribution, the single‐task backchannel prediction model fails to predict any of the backchannel categories, and the proposed multitask approach substantially enhances the backchannel prediction performance. Notably, in the streaming prediction scenario, the performance of backchannel prediction improves by up to 18.7% compared with existing methods.

show abstract

Section: Resultsmentioning

confidence: 81%

“…BCP was the focus on Ortega and others [8] and Jang and others [7]. Nevertheless, training in Ruede [9], which focused on BOP, aligned with the procedures of these two studies.…”

Section: Resultsmentioning

confidence: 99%

Section: Related Workmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

Joint streaming model for backchannel prediction and automatic speech recognition

Choi,

Bang,

Kim

2024

ETRI Journal

View full text Add to dashboard Cite

show abstract

“…Additionally, the approaches needed to exploit at least one visual cue (e.g., landmarks, image, visually annotated behavioral labels). We acknowledge though many works that use exclusively lexical or audio features to predict non-verbal social cues such as backchannel opportunities or turn-taking events (Ortega et al, 2020;Jang et al, 2021). On the other hand, the taxonomy is very flexible with regards to the typology of predicted human behavior.…”

Section: Taxonomymentioning

confidence: 99%

Didn't see that coming: a survey on non-verbal social human behavior forecasting

Barquero¹,

Núñez²,

Escalera³

et al. 2022

Preprint

View full text Add to dashboard Cite

Non-verbal social human behavior forecasting has increasingly attracted the interest of the research community in recent years. Its direct applications to human-robot interaction and socially-aware human motion generation make it a very attractive field. In this survey, we define the behavior forecasting problem for multiple interactive agents in a generic way that aims at unifying the fields of social signals prediction and human motion forecasting, traditionally separated. We hold that both problem formulations refer to the same conceptual problem, and identify many shared fundamental challenges: future stochasticity, context awareness, history exploitation, etc. We also propose a taxonomy that comprises methods published in the last 5 years in a very informative way and describes the current main concerns of the community with regard to this problem. In order to promote further research on this field, we also provide a summarized and friendly overview of audiovisual datasets featuring non-acted social interactions. Finally, we describe the most common metrics used in this task and their particular issues.

show abstract

Characteristic Comparison of Korean Unstructured Dialogue Corpora by Morphological Analysis

Moon

Shin

Kim

et al. 2022

2022 IEEE 4th International Conference on Artificial Intelligence Circuits and Systems (AICAS)

View full text Add to dashboard Cite

BPM_MT: Enhanced Backchannel Prediction Model using Multi-Task Learning

Cited by 8 publications

References 21 publications

Joint streaming model for backchannel prediction and automatic speech recognition

Joint streaming model for backchannel prediction and automatic speech recognition

Didn't see that coming: a survey on non-verbal social human behavior forecasting

Characteristic Comparison of Korean Unstructured Dialogue Corpora by Morphological Analysis

Contact Info

Product

Resources

About