Language model adaptation for video lectures transcription

Abstract

Videolectures are currently being digitised all over the world for its enormous value as reference resource. Many of these lectures are accompanied with slides. The slides offer a great opportunity for improving ASR systems performance. We propose a simple yet powerful extension to the linear interpolation of language models for adapting language models with slide information. Two types of slides are considered, correct slides, and slides automatic extracted from the videos with OCR. Furthermore, we compare both time aligned and unaligned slides. Results report an improvement of up to 3.8 % absolute WER points when using correct slides. Surprisingly, when using automatic slides obtained with poor OCR quality, the ASR system still improves up to 2.2 absolute WER points.

Language Model Adaptation for Lecture Transcription by Document Retrieval

“…We compare our approach with a strong baseline computed from a large collection of out-of-domain and in-domain documents comprising 46 billion words. Furthermore, we compare our results with those obtained by slide adaptation [15], using as slides the text extracted from the video using OCR. We also combine both approaches to further improve adaptation which yields significant improvements with respect to both the baseline model and the slide-adapted model.…”

“…In this work, we further consider the scenario where the lecture slides can be extracted from the video using OCR and they are available to adapt the models [15], or a mixed scenario that combines both the text in the slides and the retrieved documents as follows…”

“…The language model adaptation technique for video lectures was introduced in [15]. It combines out-of-domain language models, in-domain models and videospecific models by means of a linear interpolation:…”

“…For instance, it is usual that either the author does not give access to the slide document, or the repository does not keep track of such files. When slides are not available in electronic format, they can be extracted from the video recording using OCR techniques [15]. However, due to the video quality, even this is not possible in many cases.…”

“…Despite current state-of-the-art automatic speech recognition (ASR) systems are achieving continuous improvements over time, these repositories can be greatly improved through the use of specifically retrieved in-domain data. For instance, in [15] the video-dependent in-domain data is extracted/retrieved from the slides used in each video. Specifically, a general-purpose ASR system was adapted through language model interpolation from different resources (out-of-domain and in-domain,) including the text of video-dependent slides.…”