A collaborative framework for collecting Thai unknown words from the web

Haruechaiyasak, Choochart; Sangkeettrakarn, Chatchawal; Palingoon, Pornpimon; Kongyoung, Sarawoot; Damrongrat, Chaianun

doi:10.3115/1273073.1273118

Cited by 11 publications

(6 citation statements)

References 8 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The longest matching and maximum matching algorithms are plagued with problems related to dictionary size and unknown words. Researchers have attempted to cope with the unknown -for example, Haruechaiyasak and his colleagues have proposed the unknown word collecting framework [10]. Theirs is an integrative framework that extracts unrecognizable words, which are then reviewed and corrected by humans before being added to a dictionary.…”

Section: Literature Reviewmentioning

confidence: 99%

Improving Thai Word and Sentence Segmentation Using Linguistic Knowledge

Nararatwong

Kertkeidkachorn

Cooharojananone

et al. 2018

IEICE Trans. Inf. & Syst.

View full text Add to dashboard Cite

Word boundary ambiguity in word segmentation has long been a fundamental challenge within Thai language processing. The Conditional Random Fields (CRF) model is among the best-known methods to have achieved remarkably accurate segmentation. Nevertheless, current advancements appear to have left the problem of compound words unaccounted for. Compound words lose their meaning or context once segmented. Hence, we introduce a dictionary-based word-merging algorithm, which merges all kinds of compound words. Our evaluation shows that the algorithm can accomplish a high-accuracy of word segmentation, with compound words being preserved. Moreover, it can also restore some incorrectly segmented words. Another problem involving a different wordchunking approach is sentence boundary ambiguity. In tackling the problem, utilizing the part of speech (POS) of a segmented word has been found previously to help boost the accuracy of CRF-based sentence segmentation. However, not all segmented words can be tagged. Thus, we propose a POS-based word-splitting algorithm, which splits words in order to increase POS tags. We found that with more identifiable POS tags, the CRF model performs better in segmenting sentences. To demonstrate the contributions of both methods, we experimented with three of their applications. With the word merging algorithm, we found that intact compound words in the product of topic extraction can help to preserve their intended meanings, offering more precise information for human interpretation. The algorithm, together with the POS-based word-splitting algorithm, can also be used to amend word-level Thai-English translations. In addition, the word-splitting algorithm improves sentence segmentation, thus enhancing text summarization.

show abstract

Section: Literature Reviewmentioning

confidence: 99%

Improving Thai Word and Sentence Segmentation Using Linguistic Knowledge

Nararatwong

Kertkeidkachorn

Cooharojananone

et al. 2018

IEICE Trans. Inf. & Syst.

View full text Add to dashboard Cite

show abstract

“…The first is the unknown word problem. Unknown words are words which are not included in the dictionary for processing the texts [2]. The second problem is due to the ambiguity while parsing the texts.…”

Section: Introductionmentioning

confidence: 99%

A comparative study on Thai word segmentation approaches

Haruechaiyasak

Kongyoung

Dailey³

2008

2008 5th International Conference on Electrical Engineering/Electronics, Computer, Telecommunications and Information Technolog

Self Cite

View full text Add to dashboard Cite

In this paper, we analyze and compare various approaches for Thai word segmentation. The word segmentation approaches could be classified into two distinct types, dictionary based (DCB) and machine learning based (MLB). The DCB approach relies on a set of terms for parsing and segmenting input texts. Whereas the MLB approach relies on a model trained from a corpus by using machine learning techniques. We compare between two algorithms from the DCB approach: longest-matching and maximal matching, and four algorithms from the MLB approach: Naive Bayes (NB), decision tree, Support Vector Machine (SVM), and Conditional Random Field (CRF). From the experimental results, the DCB approach yielded better performance than the NB, decision tree and SVM algorithms from the MLB approach. However, the best performance was obtained from the CRF algorithm with the precision and recall of 95.79% and 94.98%, respectively.

show abstract

“…As a more recent work, Haruechaiyasak et al [19] proposed a semi-automated framework that utilized statistical and corpus-based concepts for detecting unknown words and then introduced a collaborative framework among a group of corpus builders to refine the obtained results. In the automated process, unknown word boundaries are identified using frequencies of strings.…”

Section: Previous Workmentioning

confidence: 99%

“…In the past, most previous works on Thai unknown word recognition [2], [15], [19] treated unknown word candidates independently. However, in the real situation, a set of candidates generated from an unregistered portion, should be considered dependently and treated as a group.…”

Section: Unknown Word Identificationmentioning

confidence: 99%

“…For example, the system may place an impossible word boundary between a consonant and a vowel. To resolve such obvious mistakes, recently several works [14], [19], [21]- [23] have applied a useful concept, namely a Thai Character Cluster (TCC) [5], [16], which is defined as an inseparable group of Thai characters based on the Thai writing system. Unlike word segmentation, segmenting a text into TCCs can be completely done without error and ambiguity by a small set of predefined rules.…”

Section: Unregistered Portions Detectionmentioning

confidence: 99%

See 1 more Smart Citation

A Corpus-Based Approach for Automatic Thai Unknown Word Recognition Using Boosting Techniques

TeCho

Nattee

Theeramunkong

2009

IEICE Trans. Inf. & Syst.

View full text Add to dashboard Cite

Jakkrit TECHO†a) , Student Member, Cholwich NATTEE †b) , and Thanaruk THEERAMUNKONG †c) , Members SUMMARY While classification techniques can be applied for automatic unknown word recognition in a language without word boundary, it faces with the problem of unbalanced datasets where the number of positive unknown word candidates is dominantly smaller than that of negative candidates. To solve this problem, this paper presents a corpus-based approach that introduces a so-called group-based ranking evaluation technique into ensemble learning in order to generate a sequence of classification models that later collaborate to select the most probable unknown word from multiple candidates. Given a classification model, the group-based ranking evaluation (GRE) is applied to construct a training dataset for learning the succeeding model, by weighing each of its candidates according to their ranks and correctness when the candidates of an unknown word are considered as one group. A number of experiments have been conducted on a large Thai medical text to evaluate performance of the proposed group-based ranking evaluation approach, namely V-GRE, compared to the conventional naïve Bayes classifier and our vanilla version without ensemble learning. As the result, the proposed method achieves an accuracy of 90.93±0.50% when the first rank is selected while it gains 97.26±0.26% when the top-ten candidates are considered, that is 8.45% and 6.79% improvement over the conventional record-based naïve Bayes classifier and the vanilla version. Another result on applying only best features show 93.93±0.22% and up to 98.85 ±0.15% accuracy for top-1 and top-10, respectively. They are 3.97% and 9.78% improvement over naive Bayes and the vanilla version. Finally, an error analysis is given. key words: unknown word recognition, word boundary detection, data mining, machine learning, ensemble learning

show abstract

A collaborative framework for collecting Thai unknown words from the web

Cited by 11 publications

References 8 publications

Improving Thai Word and Sentence Segmentation Using Linguistic Knowledge

Improving Thai Word and Sentence Segmentation Using Linguistic Knowledge

A comparative study on Thai word segmentation approaches

A Corpus-Based Approach for Automatic Thai Unknown Word Recognition Using Boosting Techniques

Contact Info

Product

Resources

About