Synchro-Sub, an adaptive multi-algorithm framework for real-time subtitling synchronisation of multi-type TV programmes

Abstract

Subtitles are critical elements in making TV as accessible as possible for people with hearing impairment, elderly people and other end-users that could need this kind of service. Moreover, delay in live-originated TV captions or subtitles is a significant factor in the audience’s perception of the absence of quality for the deaf and hard of hearing. This lack of synchronisation leads to poor quality broadcasts because of the inconsistency between images and subtitles presented to the viewer. In some cases, this situation leads to a misunderstanding of the message. Subtitling is a reactive process that starts once a minimum of audio information has been generated and ends when an insertion time, deletion time and text are allocated. Therefore, there will always be a delay associated with this process, related to the audio duration being subtitled and the subtitle production time. Moreover, in live TV programmes, unsynchronised audio and subtitles are inherent to the subtitling process because the broadcasting is performed in real-time. In this paper, the authors present the Synchro-Sub framework for synchronising subtitles with the audio-visual contents for live-captioned TV programs. For doing this, and considering the knowledge obtained in previous approaches defined by the authors, several improvements have been included in the transcription and chronization phases of the framework that increase the percentage of correctly timed subtitles.

Silent no more: a comprehensive review of artificial intelligence, deep learning, and machine learning in facilitating deaf and mute communication

“…By leveraging speech recognition and language processing techniques, AI systems can generate synchronized captions and subtitles for live events, videos, and other multimedia content. This feature enables individuals with hearing disabilities to understand and engage with audiovisual content, making educational materials, entertainment, and public events more inclusive (Masiello-Ruiz et al 2023). -Voice Assistants and Augmentative and Alternative Communication (AAC) Devices: AI-enabled voice assistants, including Siri, Google Assistant, and Amazon Alexa, have become increasingly prevalent and can be harnessed to aid individuals with hearing and speech impairments.…”