Recognition rate estimation based on word alignment network and discriminative error type classification

Abstract

Techniques for estimating recognition rates without using reference transcriptions are essential if we are to judge whether or not speech recognition technology is applicable to a new task. This paper proposes two recognition rate estimation methods for continuous speech recognition. The first is an easy-to-use method based on a word alignment network (WAN) obtained from a word confusion network through simple conversion procedures. A WAN contains the correct (C), substitution error (S), insertion error (I) and deletion error (D) probabilities word-by-word for a recognition result. By summing these CSID probabilities individually, the percent correct and word accuracy (WACC) can be estimated without using a reference transcription. The second more advanced method refines the CSID probabilities provided by a WAN based on discriminative error type classification (ETC) and estimates the recognition rates more accurately. In the experiments on the MIT lecture speech corpus, we obtained 0.97 of correlation coefficient between the true WACCs calculated by a scoring tool using reference transcriptions and the WACCs estimated from the discriminative ETC results.

ASR error detection and recognition rate estimation using deep bidirectional recurrent neural networks

“…Error type classification [15,23,24] can be divided into two sub tasks. The first sub task is CSI classification.…”

“…As a result, in our deletion error detection, as shown in Fig. 2, the inter-word position that precedes the t-th recognized actual word wt is labeled as "the inter-word position that has one or more deletion errors (D)" or "that has no deletion error (D)" (details are provided in [23]). As with the other tasks, P (D|wt) + P (D|wt) = 1.…”

“…Recognition rate estimation [15,23,24] can be conducted using the results of error type classification. It is an essential technique if we are to judge whether or not ASR technology is applicable to a new task at low cost, i.e.…”

“…The ASR settings were basically the same as those described in [15,23,24]. A discriminatively trained GMM-HMM-based acoustic model [29] and a word trigram with a 16.5k vocabulary size were used in the SOLON decoder [30].…”

“…We investigate three types of ASR error detection tasks, i.e. confidence estimation, OOV word detection and our proposed error type classification [15,23,24] (Section 3). We also estimate recognition rates from the error type classification results.…”