An evaluation to detect and correct erroneous characters wrongly substituted, deleted and inserted in Japanese and English sentences using Markov models

Abstract

In optical character recognition and coni.inuous speech recognition of a natural language, it has been diflicult to detect error characters which are wrongly deleted and inserted. ]n <>rder to judge three types of the errors, which are characters wrongly substituted, deleted or inserted in a Japanese "bunsetsu" and an l';nglish word, and to correct these errors, this paper proposes new methods using rn-th order Markov chain model for Japanese "l~anjikana" characters and Fmglish alphabets, assuming that Markov l)robability of a correct chain of syllables or "kanji-kana" characters is greater than that of erroneous chains. From the results of the experiments, it is concluded that the methods is usefld for detecting as well as correcting these errors in Japanese "bunsetsu" and English words.

A revised method to detect erroneous characters wrongly substituted, deleted, and inserted at the end position in Japanese sentences and ‘bunsetsu’s

“…Up to now, the methods to detect and correct erroneous characters wrongly substituted, deleted, or inserted at the inner position in Japanese sentences using m th‐order Markov chain model for Japanese ‘kanji‐kana’ characters, have been known to be useful to detect and correct these erroneous characters [11–18]. For an example, the value of the second‐order Markov probability for each character of the erroneous chain \documentclass{article}\usepackage{amsmath}\usepackage{amssymb}\usepackage{amsbsy}\usepackage{amsfonts}\pagestyle{empty}\begin{document}$\Gamma^{(2)}_S$ \end{document} or \documentclass{article}\usepackage{amsmath}\usepackage{amssymb}\usepackage{amsbsy}\usepackage{amsfonts}\pagestyle{empty}\begin{document}$\Gamma^{(2)}_I$ \end{document} remains smaller value than the critical value T just four times.…”

“…In order to solve this problem, by using the relation between the types of errors and the length of a chain in which the values of Markov joint probability remain small, a new method has been proposed to judge the three types of the errors, which are characters wrongly substituted, deleted, or inserted in Japanese sentences and ‘bunsetsu’s; to find the locations and the lengths of these erroneous characters; and to correct these errors in Japanese ‘kanji‐kana’ chains using m th‐order Markov chain model [11–18].…”

Detection and correction of mutually interfered erroneous characters in Japanese texts

“…An example of 2nd-order Markov chain models to skip one character, is shown in Fig.3. The precise definitions of the error types, the "Relevance Factor" P and the "Recall Factor" R are given in [2].…”

Japanese OCR Error Correction Using Stochastic Morphological Analyzer and Probabilistic Word N-gram Model

“…Araki et al [1] tried to correct not only substitution errors but also insertion and deletion errors in Japanese text using character trigram statistics. In insertion errors, some wrong characters are inserted into the original text, and in deletion errors, some characters are lost from the original text.…”

“…Instead, it has been shown that character n-gram statistics are effective in detecting and correcting erroneous Japanese text [1,7,8].…”