Unsupervised Neural Word Segmentation for Chinese via Segmental Language Modeling

Abstract

Previous traditional approaches to unsupervised Chinese word segmentation (CWS) can be roughly classified into discriminative and generative models. The former uses the carefully designed goodness measures for candidate segmentation, while the latter focuses on finding the optimal segmentation of the highest generative probability. However, while there exists a trivial way to extend the discriminative models into neural version by using neural language models, those of generative ones are non-trivial. In this paper, we propose the segmental language models (SLMs) for CWS. Our approach explicitly focuses on the segmental nature of Chinese, as well as preserves several properties of language models. In SLMs, a context encoder encodes the previous context and a segment decoder generates each segment incrementally. As far as we know, we are the first to propose a neural model for unsupervised CWS and achieve competitive performance to the state-of-theart statistical models on four different datasets from SIGHAN 2005 bakeoff.

Multilingual unsupervised sequence segmentation transfers to extremely low-resource languages

“…Another line of techniques have been focused on models that are both strong language models and good for sequence segmentation. Many are in some way based on Connectionist Temporal Classification (Graves et al, 2006), and include Sleep-WAke Networks (Wang et al, 2017), Segmental RNNs (Kong et al, 2016), and Segmental Language Models (Sun and Deng, 2018;Kawakami et al, 2019;Wang et al, 2021;Downey et al, 2021). In this work, we conduct experiments using the Masked Segmental Language Model of Downey et al (2021), due to its good performance and scalability, the latter usually regarded as an obligatory feature of crosslingual models (Conneau et al, 2020a;Xue et al, 2021, inter alia).…”

“…The sentences originally came in a train/validation/test split, but because goldsegmented sentences are so rare, we concatenate these sets and then split them in half into final validation and test sets. MSLMs An MSLM is a variant of a Segmental Language Model (SLM) (Sun and Deng, 2018;Kawakami et al, 2019;Wang et al, 2021), which takes as input a sequence of characters x and outputs a probability distribution for a sequence of segments y such that the concatenation of the segments of y is equivalent to x: π(y) = x. An MSLM is composed of a Segmental Transformer Encoder and an LSTM-based Segment Decoder (Downey et al, 2021).…”

“…Unsupervised sequence segmentation (at the word, morpheme, and phone level) has long been an area of interest in languages without whitespacedelimited orthography (e.g. Chinese, Uchiumi et al, 2015;Sun and Deng, 2018), morphologically complex languages without rule-based morphological anlayzers (Creutz and Lagus, 2002), and automatically phone-transcribed speech data Lane et al, 2021) respectively. It has been particularly important for lowerresource languages in which there is little or no gold-standard data on which to train supervised models.…”

A Masked Segmental Language Model for Unsupervised Natural Language Segmentation

“…To solve this problem we propose a type of Segmental Language Model (Sun and Deng, 2018;Kawakami et al, 2019), based on the powerful neural Transformer architecture (Vaswani et al, 2017).…”

“…Nearperfect supervised methods have been developed for use in resource-rich languages such as Chinese, but many of the world's languages are both morphologically complex, and have no large dataset of "gold" segmentations into meaningful units. To solve this problem, we propose a new type of Segmental Language Model (Sun and Deng, 2018;Kawakami et al, 2019;Wang et al, 2021), for use in both unsupervised and lightly supervised segmentation tasks. We introduce a Masked Segmental Language Model (MSLM) built on a spanmasking transformer architecture, harnessing the power of a bi-directional masked modeling context and attention.…”