A Hybrid Approach to Automatic Corpus Generation for Chinese Spelling Check

Abstract

Chinese spelling check (CSC) is a challenging yet meaningful task, which not only serves as a preprocessing in many natural language processing (NLP) applications, but also facilitates reading and understanding of running texts in peoples' daily lives. However, to utilize datadriven approaches for CSC, there is one major limitation that annotated corpora are not enough in applying algorithms and building models. In this paper, we propose a novel approach of constructing CSC corpus with automatically generated spelling errors, which are either visually or phonologically resembled characters, corresponding to the OCRand ASR-based methods, respectively. Upon the constructed corpus, different models are trained and evaluated for CSC with respect to three standard test sets. Experimental results demonstrate the effectiveness of the corpus, therefore confirm the validity of our approach. * This work was conducted during Dingmin Wang's internship in Tencent AI Lab. Sentence Correction 我们应该认真对待这些 己 (ji2) 经发生的事 已 (yi3) 在我们班上， 她 (ta1)是一个很聪明的男孩 他 (ta1)

FASPell: A Fast, Adaptable, Simple, Powerful Chinese Spell Checker Based On DAE-Decoder Paradigm

“…As shown in Table 6, FASPell achieves state-ofthe-art F1 performance on both detection level and correction level. It is better in precision than the model by Wang et al (2018) and better in recall than the model by Zhang et al (2015). In comparison with Zhao et al (2017), It is better by any metric.…”

“…2. insufficiency in utilizing character similarity. Since a cut-off threshold of quantified character similarity (Liu et al, 2010;Wang et al, 2018) is used to produce the confusion set, similar characters are actually treated indiscriminately in terms of their similarity. This means the information of character similarity is not sufficiently utilized.…”

“…Since Chinese spell checking data require tedious professional manual work, they have always been underresourced. To prevent the filter from overfitting, Wang et al (2018) propose an automatic method to generate pseudo spell checking data. However, the precision of their spell checking model ceases to improve when the generated data reaches 40k sentences.…”

Chunk-based Chinese Spelling Check with Global Optimization

“…Hsieh et al (2015) propose to extract spelling error samples from the Google web 1T corpus. Wang et al (2018) propose the OCR-based and ASR-based methods to mimic human errors. They further proposed a pointer network to model the CSC task under the framework of a seq2seq model .…”

“…The results with ‡ are reproduced by rerunning the released code and evaluation scripts on the standard CSC datasets. TheWang et al (2018) and calculate the performance on the character-level, which makes their results incomparable with other works.…”