Attention Is All You Need for Chinese Word Segmentation

Abstract

Taking greedy decoding algorithm as it should be, this work focuses on further strengthening the model itself for Chinese word segmentation (CWS), which results in an even more fast and more accurate CWS model. Our model consists of an attention only stacked encoder and a light enough decoder for the greedy segmentation plus two highway connections for smoother training, in which the encoder is composed of a newly proposed Transformer variant, Gaussian-masked Directional (GD) Transformer, and a biaffine attention scorer. With the effective encoder design, our model only needs to take unigram features for scoring. Our model is evaluated on SIGHAN Bakeoff benchmark datasets. The experimental results show that with the highest segmentation speed, the proposed model achieves new state-of-the-art or comparable performance against strong baselines in terms of strict closed test setting.

Enhancing Chinese Word Segmentation via Pseudo Labels for Practicability

“…Aiming at not only keeping competitive performance on benchmarks but also reducing the complexity of the CWS methods, our proposed framework consists of two essential modules: a student model and a teacher model, as shown in Figure 1. There is an obvious performance gap between the model based on PLMs and the lightweight model (Duan and Zhao, 2020). The OOV issue is the main reason for the gap.…”

“…-Transformer. This paper adopts a modified Transformer which follows the previous study by Duan and Zhao (2020). The modified Transformer changes the multi-head self-attention to the multihead Gaussian directional attention.…”

“…In particular, Long Short-Term Memory Networks (LSTM) are the main backbone networks being used in these methods Ma et al, 2018;. Except for using LSTM, self-attention networks have been also employed for CWS (Duan and Zhao, 2020).…”

“…The Transformer is usually not working as well as LSTM for sequence labeling tasks despite its success on other tasks. We propose a new Transformer variant that is inspired by Duan and Zhao (2020). The modified Transformer utilizes the Gaussian directional mask to encode unigram features.…”

“…We utilize the uniform-sample to choose the hyper-parameters. In particular, we use the hyper-parameters of the modified Transformer model and the NMT model following previous studies (Vaswani et al, 2017;Duan and Zhao, 2020).…”