Spelling Error Correction with Soft-Masked BERT

Abstract

Spelling error correction is an important yet challenging task because a satisfactory solution of it essentially needs human-level language understanding ability. Without loss of generality we consider Chinese spelling error correction (CSC) in this paper. A state-ofthe-art method for the task selects a character from a list of candidates for correction (including non-correction) at each position of the sentence on the basis of BERT, the language representation model. The accuracy of the method can be sub-optimal, however, because BERT does not have sufficient capability to detect whether there is an error at each position, apparently due to the way of pre-training it using mask language modeling. In this work, we propose a novel neural architecture to address the aforementioned issue, which consists of a network for error detection and a network for error correction based on BERT, with the former being connected to the latter with what we call soft-masking technique. Our method of using 'Soft-Masked BERT' is general, and it may be employed in other language detectioncorrection problems. Experimental results on two datasets demonstrate that the performance of our proposed method is significantly better than the baselines including the one solely based on BERT.

Chunk-based Chinese Spelling Check with Global Optimization

“…In the experiment, our method outperforms Hong et al (2019) with a large margin, which indicates the effectiveness of the globally optimized chunk-based decoding. Zhang et al (2020) propose to train a detection and a correction network jointly. In the experiment, although they employ 5 million pseudo data for extra pretraining, the proposed method still obtains an improved performance on the correction level.…”

“…generate pseudo data by replacing the character in the training sentence with characters in the confusion set. Similarly, Zhang et al (2020) generate homophonous pseudo data to pretrain the detection and correction network jointly. Web texts are in large quantities and contain more errors than published articles.…”

“…The accumulated errors make the spell checking even more difficult. Thus, characterbased models are proposed to perform the correction at the character-level directly, which are more robust to segmentation errors (Zhang et al, 2015;Hong et al, 2019;Zhang et al, 2020). However, the character-based model cannot effectively utilize the word-level semantic information, and the correction is also more difficult to interpret.…”

Correcting Chinese Spelling Errors with Phonetic Pre-training

“…Traditional methods of CSC firstly detect misspelled characters and generate candidates via a language model, and then use a phonetic model or rules to filter wrong candidates (Chang, 1995;Chen et al, 2013;Dong et al, 2016). To improve CSC performance, studies mainly focus on two issues: 1) how to improve the language model (Wu et al, 2010;Dong et al, 2016;Zhang et al, 2020) and 2) how to utilize external knowledge of phonological similarity (Jia et al, 2013;Yu and Li, 2014;Cheng et al, 2020). The language model is used to generate fluent sentences and the phonetic features can prevent the model from producing predictions whose pronunciation deviates from that of the original word.…”

“…These methods take phonetic information as external knowledge but the discrete candidate selection obstructs the language model from learning directly via backpropagation. Zhang et al (2020) proposed an endto-end CSC model by modifying the mask mechanism of BERT. However, they did not use any phonological information, which is important for exploring words similarity.…”