Automatic dubbing of TV programmes for the hearing impaired

Abstract

Abstract-This paper presents experiments with a customisation of a corpus-based unit-selection text-to-speech (TTS) system for automatic dubbing of TV programmes. The project aims at people with hearing impairments as its main goal is to produce a highly intelligible, less-dynamic, and more-undisturbed audio track for TV programmes automatically from subtitles. A twophase synchronisation process was proposed to cope with audiovideo synchronisation issues. These phases include both off-line time compression of all utterances in a source speech corpus used for TTS and on-line time compression of speech that overlaps assigned subtitle time slots. Based on a case study, in which a TTS-generated audio track of a selected movie was analysed, a simplification of to-be-desynchronised subtitle texts was proposed in order to keep time-compression factors in a reasonable extent. In this way, abrupt changes in dynamics of the produced audio track are avoided.

Prosodic Phrase Alignment for Machine Dubbing

“…A perception test has also been prepared to compare our methodology to a subtitle reading approach similar to [4,5,6]. We have selected 10 samples from our evaluation set and cut out the video portions corresponding to them.…”

“…Automatic dubbing has been previously proposed in various works without any translation back-end [4,5,6,7]. General motivation behind these works is to generate an additional track for a given audiovisual for aiding viewers with auditory or visual impairment.…”

“…This is obtained by placing the synthesized segment at the beginning of each subtitle entry and then adjusting its speech rate only if it exceeds the duration of the subtitle entry. Matousek et al performs an online and offline TTS time compression technique in order to achieve this [5]. Although this methodology can be useful in news or documentary domain, it would lead to limitations for obtaining realistically synthesized segments needed in e.g.…”

Proceedings of the 17th International Conference on Spoken Language Translation

“…While STST has been addressed for long time and by several research labs (Waibel, 1996;Vidal, 1997;Metze et al, 2002;Nakamura et al, 2006;Wahlster, 2013), relatively less and more sparse efforts have been devoted to automatic dubbing (Matousek et al, 2010;Matousek and Vít, 2012;Furukawa et al, 2016;Öktem et al, 2019), although the potential demand of such technology could be huge. In fact, multimedia content created and put online has been growing at exponential rate, in the last decade, while availability and cost of human skills for subtitling and dubbing still remains a barrier for its diffusion worldwide.…”

Automatic fingersign-to-speech translation system

“…They have been developed in recent years i.e. faster unit selection [10], expressive speech synthesis [19] or application [40].…”