Frame-Synchronous and Local Confidence Measures for Automatic Speech Recognition

Abstract

In this paper, we introduce two new confidence measures for large vocabulary speech recognition systems. The major feature of these measures is that they can be computed without waiting for the end of the audio stream. We proposed two kinds of confidence measures: frame-synchronous and local. The frame-synchronous ones can be computed as soon as a frame is processed by the recognition engine and are based on a likelihood ratio. The local measures estimate a local posterior probability in the vicinity of the word to analyze. We evaluated our confidence measures within the framework of the automatic transcription of French broadcast news with the EER criterion. Our local measures achieved results very close to the best state-of-the-art measure (EER of 23% compared to 22.0%). We then conducted a preliminary experiment to assess the contribution of our confidence measure in improving the comprehension of an automatic transcription for the hearing impaired. We introduced several modalities to highlight words of low confidence in this transcription. We showed that these modalities used with our local confidence measure improved the comprehension of automatic transcription.

Qualitative investigation of the display of speech recognition results for communication with deaf people

“…These results are similar to those showed from a previous study undertaken among a hearing population [15].…”

“…In all the previous studies, the speech signal was available to the user. This is not the case of [15] which has investigated the understanding of sentences from their speech recognition output only, and investigated how much taking into account the confidence measures in the display can help.…”

“…In [15], it has been shown that hearing users infer the correct word from a word considered incorrect by the speech recognition system, more easily when it was written in phonetic form than when it was written in orthographic form. In particular, when several consecutive words were tagged as misrecognized by the system, the hearing user unsuccessfully focused on the word splitting given by the orthographic mode, causing misunderstandings, while the sound sequence of the words was almost free from errors.…”

Confidence measure by substring comparison for automatic speech recognition

“…In some scenarios it is very important to compute confidence measures without waiting for the end of the audio stream [7]. The frame-synchronous ones can be computed as soon as a frame is processed by the recognition system and are based on a likelihood ratio.…”

“…Existing types and applications of confidence measures were well summarised [5][6][7]. Confidence measures can help to decide to keep or reject a hypothesis in keyword spotting applications.…”