Simultaneous Synchronization of Text and Speech for Broadcast News Subtitling

Gao, Jie; Zhao, Qingwei; Ta, Li; Yan, Yonghong

doi:10.1007/978-3-642-01513-7_63

Cited by 6 publications

(11 citation statements)

References 7 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Since there is time enough between the production and the broadcast of the audio-visual content, it is possible to carefully produce the subtitles and review them ''before the broadcast''. In a manual or semi-automatic way locutions may be transcribed, split them in subtitles and synchronize them with the audio [4].…”

Section: A Pre-recorded Programsmentioning

confidence: 99%

“…Usually, a person manually inserts the subtitles following the pace of the speakers. For programs with simple environments such as news programs, the literature offers a wide range of techniques for the automation of synchronization [4], [15]- [17].…”

Section: B Live-scripted Programsmentioning

confidence: 99%

“…In these circumstances, it is not possible to know in advance what the necessary subtitles will be, so they will have to be generated in real time ''during the broadcast'' [4]. This necessarily implies a loss of quality, as the systems that produce them: human, machine or mixed, are not error-free and the possibility of correcting them is very limited in time.…”

Section: Live-improvised Programsmentioning

confidence: 99%

“…For live programs, such as broadcast news, the ratio is even lower. Reference [4] presents a system for subtitling news programs on Chinese television. The system is formed by two main modules.…”

Section: Literature Reviewmentioning

confidence: 99%

“…Taking into account the world disability report [3], there are over one thousand million people with at least one type of disability. Accordingly, there is increasing demand by individuals and associations of disabled persons to make cultural activities accessible, and in particular, for television programme contents to be subtitled, and furthermore, for this subtitling to be of high-quality [4]. This situation has led governments and regulatory entities to establish mandatory requirements for broadcasters to ensure that television has subtitling and sign language services for the deaf,and audio description services for the blind or visually-impaired [5].…”

Section: Introductionmentioning

confidence: 99%

See 4 more Smart Citations

Sub-Sync: Automatic Synchronization of Subtitles in the Broadcasting of True Live programs in Spanish

González‐Carrasco

Puente²,

Ruíz‐Mezcua

et al. 2019

IEEE Access

View full text Add to dashboard Cite

Individuals with sensory impairment (hearing or visual) encounter serious communication barriers within society and the world around them. These barriers hinder the communication process and make access to information an obstacle they must overcome on a daily basis. In this context, one of the most common complaints made by the Television (TV) users with sensory impairment is the lack of synchronism between audio and subtitles in some types of programs. In addition, synchronization remains one of the most significant factors in audience perception of quality in live-originated TV subtitles for the deaf and hard of hearing. This paper introduces the Sub-Sync framework intended for use in automatic synchronization of audiovisual contents and subtitles, taking advantage of current well-known techniques used in symbol sequences alignment. In this particular case, these symbol sequences are the subtitles produced by the broadcaster subtitling system and the word flow generated by an automatic speech recognizing the procedure. The goal of Sub-Sync is to address the lack of synchronism that occurs in the subtitles when produced during the broadcast of live TV programs or other programs that have some improvised parts. Furthermore, it also aims to resolve the problematic interphase of synchronized and unsynchronized parts of mixed type programs. In addition, the framework is able to synchronize the subtitles even when they do not correspond literally to the original audio and/or the audio cannot be completely transcribed by an automatic process. Sub-Sync has been successfully tested in different live broadcasts, including mixed programs, in which the synchronized parts (recorded, scripted) are interspersed with desynchronized (improvised) ones. INDEX TERMS Accessibility, TV broadcasting, algorithm design and analysis, automatic speech recognition.

show abstract

Section: A Pre-recorded Programsmentioning

confidence: 99%

Section: B Live-scripted Programsmentioning

confidence: 99%

Section: Live-improvised Programsmentioning

confidence: 99%

Section: Literature Reviewmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 3 more Smart Citations

Sub-Sync: Automatic Synchronization of Subtitles in the Broadcasting of True Live programs in Spanish

González‐Carrasco

Puente²,

Ruíz‐Mezcua

et al. 2019

IEEE Access

View full text Add to dashboard Cite

show abstract

Deep-Sync: A novel deep learning-based tool for semantic-aware subtitling synchronisation

Martín

González‐Carrasco

Rodríguez-Fernández

et al. 2021

Neural Comput & Applic

View full text Add to dashboard Cite

Subtitles are a key element to make any media content accessible for people who suffer from hearing impairment and for elderly people, but also useful when watching TV in a noisy environment or learning new languages. Most of the time, subtitles are generated manually in advance, building a verbatim and synchronised transcription of the audio. However, in TV live broadcasts, captions are created in real time by a re-speaker with the help of a voice recognition software, which inevitability leads to delays and lack of synchronisation. In this paper, we present Deep-Sync, a tool for the alignment of subtitles with the audio-visual content. The architecture integrates a deep language representation model and a real-time voice recognition software to build a semantic-aware alignment tool that successfully aligns most of the subtitles even when there is no direct correspondence between the re-speaker and the audio content. In order to avoid any kind of censorship, Deep-Sync can be deployed directly on users' TVs causing a small delay to perform the alignment, but avoiding to delay the signal at the broadcaster station. Deep-Sync was compared with other subtitles alignment tool, showing that our proposal is able to improve the synchronisation in all tested cases.

show abstract

Synchro-Sub, an adaptive multi-algorithm framework for real-time subtitling synchronisation of multi-type TV programmes

et al. 2023

View full text Add to dashboard Cite

Subtitles are critical elements in making TV as accessible as possible for people with hearing impairment, elderly people and other end-users that could need this kind of service. Moreover, delay in live-originated TV captions or subtitles is a significant factor in the audience’s perception of the absence of quality for the deaf and hard of hearing. This lack of synchronisation leads to poor quality broadcasts because of the inconsistency between images and subtitles presented to the viewer. In some cases, this situation leads to a misunderstanding of the message. Subtitling is a reactive process that starts once a minimum of audio information has been generated and ends when an insertion time, deletion time and text are allocated. Therefore, there will always be a delay associated with this process, related to the audio duration being subtitled and the subtitle production time. Moreover, in live TV programmes, unsynchronised audio and subtitles are inherent to the subtitling process because the broadcasting is performed in real-time. In this paper, the authors present the Synchro-Sub framework for synchronising subtitles with the audio-visual contents for live-captioned TV programs. For doing this, and considering the knowledge obtained in previous approaches defined by the authors, several improvements have been included in the transcription and chronization phases of the framework that increase the percentage of correctly timed subtitles.

show abstract

Simultaneous Synchronization of Text and Speech for Broadcast News Subtitling

Cited by 6 publications

References 7 publications

Sub-Sync: Automatic Synchronization of Subtitles in the Broadcasting of True Live programs in Spanish

Sub-Sync: Automatic Synchronization of Subtitles in the Broadcasting of True Live programs in Spanish

Deep-Sync: A novel deep learning-based tool for semantic-aware subtitling synchronisation

Synchro-Sub, an adaptive multi-algorithm framework for real-time subtitling synchronisation of multi-type TV programmes

Contact Info

Product

Resources

About