Stream decoding for simultaneous spoken language translation

Kolss, Muntsin; Vogel, Stephan; Waibel, Alex

doi:10.21437/interspeech.2008-678

Cited by 8 publications

(4 citation statements)

References 9 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Although these systems can benefit any student, they may enhance accessibility for students with special needs through text-to-speech and vice versa. While certain services enhance lectures with transcriptions [20,21] or translations [22] of what the lecturer has said, others focus on facilitating conversational interactions between students and the system. These conversational interactions are transforming education by improving communication, engagement, and personalization for students and educators alike.…”

Section: Chatbots For Education and Learningmentioning

confidence: 99%

Knowledge-Based and Generative-AI-Driven Pedagogical Conversational Agents: A Comparative Study of Grice’s Cooperative Principles and Trust

Wölfel,

Shirzad,

Reich

et al. 2023

BDCC

View full text Add to dashboard Cite

The emergence of generative language models (GLMs), such as OpenAI’s ChatGPT, is changing the way we communicate with computers and has a major impact on the educational landscape. While GLMs have great potential to support education, their use is not unproblematic, as they suffer from hallucinations and misinformation. In this paper, we investigate how a very limited amount of domain-specific data, from lecture slides and transcripts, can be used to build knowledge-based and generative educational chatbots. We found that knowledge-based chatbots allow full control over the system’s response but lack the verbosity and flexibility of GLMs. The answers provided by GLMs are more trustworthy and offer greater flexibility, but their correctness cannot be guaranteed. Adapting GLMs to domain-specific data trades flexibility for correctness.

show abstract

Section: Chatbots For Education and Learningmentioning

confidence: 99%

Knowledge-Based and Generative-AI-Driven Pedagogical Conversational Agents: A Comparative Study of Grice’s Cooperative Principles and Trust

Wölfel,

Shirzad,

Reich

et al. 2023

BDCC

View full text Add to dashboard Cite

show abstract

“…Simultaneous speech translation can be implemented either as a (hybrid) cascaded system (Kolss et al, 2008;Elbayad et al, 2020;Liu et al, 2020a;Bahar et al, 2021) or an end-to-end model Liu et al, 2021). Unlike for the offline speech translation where cascade seems to have the best quality, the end-to-end speech translation offers a better qualitylatency trade-off (Ansari et al, 2020;Liu et al, 2021;Anastasopoulos et al, 2021).…”

Section: Related Workmentioning

confidence: 99%

CUNI-KIT System for Simultaneous Speech Translation Task at IWSLT 2022

Polák¹,

Pham²,

Nguyen³

et al. 2022

Proceedings of the 19th International Conference on Spoken Language Translation (IWSLT 2022)

View full text Add to dashboard Cite

In this paper, we describe our submission to the Simultaneous Speech Translation at IWSLT 2022. We explore strategies to utilize an offline model in a simultaneous setting without the need to modify the original model. In our experiments, we show that our onlinization algorithm is almost on par with the offline setting while being 3× faster than offline in terms of latency on the test set. We also show that the onlinized offline model outperforms the best IWSLT2021 simultaneous system in medium and high latency regimes and is almost on par in the low latency regime. We make our system publicly available. 1

show abstract

“…Speech-to-Speech translation systems have matured in recent years from early prototypes over mobile hand-held translators to fully integrated and operational simultaneous interpreting systems that have been deployed in lecture and video conferencing applications [14,16,26,31,36,62,63].…”

Section: Introductionmentioning

confidence: 99%

Face-Dubbing++: LIP-Synchronous, Voice Preserving Translation Of Videos

Waibel

Behr

Yaman

et al. 2023

2023 IEEE International Conference on Acoustics, Speech, and Signal Processing Workshops (ICASSPW)

View full text Add to dashboard Cite

In this paper, we propose a neural end-to-end system for voice preserving, lip-synchronous translation of videos. The system is designed to combine multiple component models and produces a video of the original speaker speaking in the target language that is lip-synchronous with the target speech, yet maintains emphases in speech, voice characteristics, face video of the original speaker. The pipeline starts with automatic speech recognition including emphasis detection, followed by a translation model. The translated text is then synthesized by a Text-to-Speech model that recreates the original emphases mapped from the original sentence. The resulting synthetic voice is then mapped back to the original speakers' voice using a voice conversion model. Finally, to synchronize the lips of the speaker with the translated audio, a conditional generative adversarial networkbased model generates frames of adapted lip movements with respect to the input face image as well as the output of the voice conversion model. In the end, the system combines the generated video with the converted audio to produce the final output. The result is a video of a speaker speaking in another language without actually knowing it. To evaluate our design, we present a user study of the complete system as well as separate evaluations of the single components. Since there is no available dataset to evaluate our whole system, we collect a test set and evaluate our system on this test set. The results indicate that our system is able to generate convincing videos of the original speaker speaking the target language while preserving the original speaker's characteristics. The collected dataset will be shared.

show abstract

Stream decoding for simultaneous spoken language translation

Cited by 8 publications

References 9 publications

Knowledge-Based and Generative-AI-Driven Pedagogical Conversational Agents: A Comparative Study of Grice’s Cooperative Principles and Trust

Knowledge-Based and Generative-AI-Driven Pedagogical Conversational Agents: A Comparative Study of Grice’s Cooperative Principles and Trust

CUNI-KIT System for Simultaneous Speech Translation Task at IWSLT 2022

Face-Dubbing++: LIP-Synchronous, Voice Preserving Translation Of Videos

Contact Info

Product

Resources

About