Sub-Sync: Automatic Synchronization of Subtitles in the Broadcasting of True Live programs in Spanish

Abstract

Individuals with sensory impairment (hearing or visual) encounter serious communication barriers within society and the world around them. These barriers hinder the communication process and make access to information an obstacle they must overcome on a daily basis. In this context, one of the most common complaints made by the Television (TV) users with sensory impairment is the lack of synchronism between audio and subtitles in some types of programs. In addition, synchronization remains one of the most significant factors in audience perception of quality in live-originated TV subtitles for the deaf and hard of hearing. This paper introduces the Sub-Sync framework intended for use in automatic synchronization of audiovisual contents and subtitles, taking advantage of current well-known techniques used in symbol sequences alignment. In this particular case, these symbol sequences are the subtitles produced by the broadcaster subtitling system and the word flow generated by an automatic speech recognizing the procedure. The goal of Sub-Sync is to address the lack of synchronism that occurs in the subtitles when produced during the broadcast of live TV programs or other programs that have some improvised parts. Furthermore, it also aims to resolve the problematic interphase of synchronized and unsynchronized parts of mixed type programs. In addition, the framework is able to synchronize the subtitles even when they do not correspond literally to the original audio and/or the audio cannot be completely transcribed by an automatic process. Sub-Sync has been successfully tested in different live broadcasts, including mixed programs, in which the synchronized parts (recorded, scripted) are interspersed with desynchronized (improvised) ones. INDEX TERMS Accessibility, TV broadcasting, algorithm design and analysis, automatic speech recognition.

A new system for automatic analysis and quality adjustment in audiovisual subtitled‐based contents by means of genetic algorithms

“…The subtitles should be perfectly synchronized with all the audio information of the content (dialogue, sound effects, or songs). This indicator only applies for deferred contents, because currently the technology presents several issues for the synchronization in live events (González-Carrasco et al, 2019). Regarding speed, one quality indicator is that any subtitle, regardless of its length, must have a duration of at least 1 s. Nowadays, language-saving techniques are used to achieve the 15 cps stipulated as quality speed.…”

“…Also, synchronization helps understanding subtitle text regarding the image, especially in news programmes (or similar), where audiovisual content is shown at such speed that if synchronization is not maintained, the subtitles may appear on screen with other completely different visual information, generating confusion among users. At the Spanish level, the "Follow-up report of the subtitles and audio description in TV" (Ruiz et al, 2014) states that one of the main complaints made by TV users is the lack of synchronism between audio and subtitles in some types of programmes (González-Carrasco et al, 2019). Moreover, delay remains one of the most significant factors in the audience's perception of lack of quality in live-originated TV captions or subtitles for the deaf and hearing-impaired (Simpson, Barrett, Bell, & Renals, 2016).…”

“…Specifically, the automatic synchronization of subtitles and audio has been examined by a significant number of studies in this area. These synchronization proposals are commonly considered a component‐tool for the process of automatic subtitle generation (González‐Carrasco et al, 2019). For programmes with simple environments such as news programmes, the literature offers a wide range of techniques for the automation of synchronization (Ando et al, 2003; Gao, Zhao, Li, & Yan, 2009; Garcia et al, 2009; Lertwongkhanakool, Punyabukkana, & Suchato, 2013).…”

“…), and their knowledge of them could help sensitize and prepare them for possible age‐associated hearing loss. Furthermore, using this technology for distribution in television or streaming on the Internet can be a key contributor to the education sector (González‐Carrasco et al, 2019).…”

“…On the one hand, the delay in the availability of subtitles is very disturbing for the consumer of audiovisual content. In this sense, synchronization in live broadcasting is a problem dealt with by the authors in another research study (González-Carrasco, Puente, Ruiz-Mezcua, & López-Cuadrado, 2019). At the same time, fidelity can be measured by the NER model, a tool widely used in the quality measurement of subtitles in live broadcasting, where the accuracy has to be checked (Romero-Fresco & Pérez, 2015).…”