SpellGCN: Incorporating Phonological and Visual Similarities into Language Models for Chinese Spelling Check

Abstract

Chinese Spelling Check (CSC) is a task to detect and correct spelling errors in Chinese natural language. Existing methods have made attempts to incorporate the similarity knowledge between Chinese characters. However, they take the similarity knowledge as either an external input resource or just heuristic rules. This paper proposes to incorporate phonological and visual similarity knowledge into language models for CSC via a specialized graph convolutional network (SpellGCN). The model builds a graph over the characters, and SpellGCN is learned to map this graph into a set of inter-dependent character classifiers. These classifiers are applied to the representations extracted by another network, such as BERT, enabling the whole network to be end-to-end trainable. Experiments 1 are conducted on three human-annotated datasets. Our method achieves superior performance against previous models by a large margin.

Chunk-based Chinese Spelling Check with Global Optimization

“…In the experiment, although they employ 5 million pseudo data for extra pretraining, the proposed method still obtains an improved performance on the correction level. Cheng et al (2020) propose to incorporate phonological and visual confusion sets into the CSC models through a graph convolutional network. As the performance reported in their paper is obtrained with external training data, we reproduced their results on the standard CSC datasets by rerunning their released code and evaluation scripts.…”

“…Zhao et al (2017) use conditional random fields (CRFs) to handle two types of misspelled single-character word. Cheng et al (2020) propose to incorporate phonological and visual similarity knowledge into the CSC models via a graph convolutional network.…”

Correcting Chinese Spelling Errors with Phonetic Pre-training

“…For the pre-training corpus, we collect a variety of data, such as encyclopedia articles, news, scientific papers, and movie subtitles from a search engine. The CSC training data used in our experiments is the same as Wang et al (2019) and Cheng et al (2020), including three human-annotated training datasets Tseng et al, 2015) and an automatically generated dataset with the approach proposed in Table 1: The performance on SIGHAN13, SIGHAN14, and SIGHAN15 testset. Soft-Masked BERT* is our reproduction of Soft-Masked BERT using the same training data as in our method, while Soft-Masked BERT was trained on an in-house dataset containing 5 million sentences and their counterparts with automatically generated errors, as reported in Zhang et al (2020), where the authors only provided their results on SIGHAN15.…”

“…• SpellGCN (Cheng et al, 2020) incorporates two similarity graphs into a pre-trained sequence-labeling model via graph convolutional network. The two graphs are derived from a confusion set and correspond to pronunciation and shape similarities.…”

“…Therefore, sentence-level evaluation is stricter and results in lower scores. Following Cheng et al (2020), we use the scripts in (Hong et al, 2019) to calculate the sentence-level results.…”