Improving Thai Word and Sentence Segmentation Using Linguistic Knowledge

Abstract

Word boundary ambiguity in word segmentation has long been a fundamental challenge within Thai language processing. The Conditional Random Fields (CRF) model is among the best-known methods to have achieved remarkably accurate segmentation. Nevertheless, current advancements appear to have left the problem of compound words unaccounted for. Compound words lose their meaning or context once segmented. Hence, we introduce a dictionary-based word-merging algorithm, which merges all kinds of compound words. Our evaluation shows that the algorithm can accomplish a high-accuracy of word segmentation, with compound words being preserved. Moreover, it can also restore some incorrectly segmented words. Another problem involving a different wordchunking approach is sentence boundary ambiguity. In tackling the problem, utilizing the part of speech (POS) of a segmented word has been found previously to help boost the accuracy of CRF-based sentence segmentation. However, not all segmented words can be tagged. Thus, we propose a POS-based word-splitting algorithm, which splits words in order to increase POS tags. We found that with more identifiable POS tags, the CRF model performs better in segmenting sentences. To demonstrate the contributions of both methods, we experimented with three of their applications. With the word merging algorithm, we found that intact compound words in the product of topic extraction can help to preserve their intended meanings, offering more precise information for human interpretation. The algorithm, together with the POS-based word-splitting algorithm, can also be used to amend word-level Thai-English translations. In addition, the word-splitting algorithm improves sentence segmentation, thus enhancing text summarization.

Character-based ThaiWord Segmentation with Multiple Attentions

“…In the early stage of Thai word segmentation, dictionary-based learning techniques had been used along with machine-learning techniques, for instance, Markov models (Kawtrakul and Thumkanon, 1997), decision trees Theeramunkong and Usanavasin, 2001), and CRFs (Haruechaiyasak et al, 2008). CRFs have been shown to be particularly suitable for Thai sequence-labeling tasks (Kruengkrai et al, 2006;Haruechaiyasak and Kongyoung, 2009;Kruengkrai et al, 2009;Nararatwong et al, 2018).…”

“…In parallel with CRFs, neural-network models, e.g., CNNs (Kittinaradorn et al, 2019;Chormai et al, 2019), LSTM (Treeratpituk, 2017), and BiL-STM (Jousimo et al, 2017), have been applied and performed excellently for character-based Thai word segmentation. Using additional knowledge, such as CC (Lapjaturapit et al, 2018;Nararatwong et al, 2018), transfer learning (Seeha et al, 2020), and stacking ensemble (Limkonchotiwat et al, 2020), along with neural-network models could improve performance.…”

“…We propose a character-based Thai wordsegmentation model with multiple attentions that jointly uses corresponding words and CCs. Our model is based on the bidirectional long short-term memory with conditional random field (BiLSTM-CRF) architecture, which is the baseline model for this study, because it has been successfully applied in sequence-labeling tasks for Thai and other languages (Jousimo et al, 2017;Nararatwong et al, 2018;Higashiyama et al, 2019;Seeha et al, 2020;Tian et al, 2020). Our contributions are as follows:…”

Semi-supervised Thai Sentence Segmentation Using Local and Distant Word Representations

“…The CRF-based model [15], which is extracted from n-grams around the considered word, achieves a F1 score of 91.9%, which is approximately 10% higher than the F1 scores achieved by other models [11,12,13] on the Orchid dataset, as mentioned in [14]. Nararatwong R. et al [45] extend this model using a POS-based word-splitting algorithm to increase identifiable POS tags, resulting in better model accuracy. Because the focus of this work is adjusting the POS as a postprocessing method, which is an input of the model instead of proposing a new sentence segmentation model, this work will not be considered in this paper.…”

Domain Adaptation of Thai Word Segmentation Models using Stacked Ensemble

“…However, unlike Chinese and Japanese, Thai WS did not receive much research attention. There are only six notable publications (Chormai et al, 2019;Nararatwong et al, 2018;Noyunsan et al;Thanadechteemapat and Fung;Tongtep and Theeramunkong) on Thai WS for the past ten years. On the other hand, there are at least eight papers from well-established conferences on Chinese and Japanese WS (Li et al, 2019;Aguirre and Aguiar, 2019;Ma et al, 2018;Gong et al, 2017;Chen et al, 2017;Zhou et al, 2017;Cai et al, 2017) within only the last two years.…”