Two Knives Cut Better Than One: Chinese Word Segmentation with Dual Decomposition

Wang, Mengqiu; Voigt, Rob; Manning, Christopher D.

doi:10.3115/v1/p14-2032

Cited by 14 publications

(7 citation statements)

References 12 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…We show that our model can accommodate the integration of both types of features. This is similar in spirit to the work of Sun (2010) and Wang et al (2014), who integrated features of character-based and word-based segmentors.…”

Section: Error Analysissupporting

confidence: 60%

Transition-Based Neural Word Segmentation

Zhang

2016

Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)

109

100

View full text Add to dashboard Cite

Character-based and word-based methods are two main types of statistical models for Chinese word segmentation, the former exploiting sequence labeling models over characters and the latter typically exploiting a transition-based model, with the advantages that word-level features can be easily utilized. Neural models have been exploited for character-based Chinese word segmentation, giving high accuracies by making use of external character embeddings, yet requiring less feature engineering. In this paper, we study a neural model for word-based Chinese word segmentation, by replacing the manuallydesigned discrete features with neural features in a word-based segmentation framework. Experimental results demonstrate that word features lead to comparable performances to the best systems in the literature, and a further combination of discrete and neural features gives top accuracies.

show abstract

Section: Error Analysissupporting

confidence: 60%

Transition-Based Neural Word Segmentation

Zhang

2016

Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)

109

100

View full text Add to dashboard Cite

show abstract

“…Further, lattice parsing-based approaches, where the input search space was represented as word-level lattices, were incorporated into CRFs (Smith, Smith, and Tromble 2005) for joint modeling of these tasks (Kudo, Yamamoto, and Matsumoto 2004). Neural sequence labeling approaches currently achieve state-of-the-art performance in word segmentation, especially for Chinese, Korean, Japanese, and so forth (Wang, Voigt, and Manning 2014;Shao, Hardmeier, and Nivre 2018). Similarly, higher-order CRFs (Müller, Schmid, and Schütze 2013) and neural morphological taggers (Malaviya, Gormley, and Neubig 2018;Tkachenko and Sirts 2018) are widely used for morphological parsing.…”

Section: Related Workmentioning

confidence: 99%

A Graph-Based Framework for Structured Prediction Tasks in Sanskrit

Krishna

Santra

Gupta

et al. 2021

Computational Linguistics

View full text Add to dashboard Cite

We propose a framework using Energy Based Models for multiple structured prediction tasks in Sanskrit. Ours is an arc-factored model, similar to the graph based parsing approaches, and we consider the tasks of word-segmentation, morphological parsing, dependency parsing, syntactic linearisation and prosodification, a prosody level task we introduce in this work. Ours is a search based structured prediction framework, which expects a graph as input, where relevant linguistic information is encoded in the nodes, and the edges are then used to indicate the association between these nodes. Typically the state of the art models for morphosyntactic tasks in morphologically rich languages still rely on hand-crafted features for their performance. But here, we automate the learning of the feature function. The feature function so learnt along with the search space we construct, encode relevant linguistic information for the tasks we consider. This enables us to substantially reduce the training data requirements to as low as 10 % as compared to the data requirements for the neural state of the art models. Our experiments in Czech and Sanskrit show the language agnostic nature of the framework, where we train highly competitive models for both the languages. Moreover, our framework enables to incorporate language specific constraints to prune the search space and to filter the candidates during inference. We obtain significant improvements in morphosyntactic tasks for Sanskrit by incorporating language specific constraints into the model. In all the tasks we discuss for Sanskrit, we either achieve state of the art results or ours is the only data driven solution for those tasks.

show abstract

“…Both of them have been followed by neural model versions (Liu et al, 2016) and (Zhang et al, 2016) respectively. There are also works integrating both character-based and word-based segmenters Sun, 2010;Wang et al, 2014) and semisupervised learning Kit, 2008b, 2011;Zeng et al, 2013;Zhang et al, 2013).…”

Section: Related Workmentioning

confidence: 99%

Fast and Accurate Neural Word Segmentation for Chinese

Cai¹,

Zhao²,

Zhang³

et al. 2017

Proceedings of the 55th Annual Meeting of the Association For Computational Linguistics (Volume 2: Short Papers)

View full text Add to dashboard Cite

Neural models with minimal feature engineering have achieved competitive performance against traditional methods for the task of Chinese word segmentation. However, both training and working procedures of the current neural models are computationally inefficient. This paper presents a greedy neural word segmenter with balanced word and character embedding inputs to alleviate the existing drawbacks. Our segmenter is truly end-toend, capable of performing segmentation much faster and even more accurate than state-of-the-art neural models on Chinese benchmark datasets.

show abstract

Two Knives Cut Better Than One: Chinese Word Segmentation with Dual Decomposition

Cited by 14 publications

References 12 publications

Transition-Based Neural Word Segmentation

Transition-Based Neural Word Segmentation

A Graph-Based Framework for Structured Prediction Tasks in Sanskrit

Fast and Accurate Neural Word Segmentation for Chinese

Contact Info

Product

Resources

About