A Hybrid Approach to Word Segmentation of Vietnamese Texts

Phương, Lê Hồng; Huyen, Nguyen Thi Minh; Roussanaly, Azim; Vinh, Ho Tuong

doi:10.1007/978-3-540-88282-4_23

Cited by 92 publications

(31 citation statements)

References 5 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…For word segmentation, we used vnTokenizer, a highly accurate segmenter which uses a hybrid approach to automatically tokenize Vietnamese text. This approach combines a finite-state automata technique, regular expression parsing, and a maximal-matching strategy augmented by statistical methods that resolve ambiguities of segmentation Phuong et al (2008). We also used JVnTagger, a POS tagger based on Conditional Random Fields Lafferty et al (2001) and Maximum Entropy Berger et al (1996).…”

Section: Toolsmentioning

confidence: 99%

Vietnamese treebank construction and entropy-based error detection

Nguyen

et al. 2015

Lang Resources & Evaluation

View full text Add to dashboard Cite

Treebanks, especially the Penn treebank for natural language processing (NLP) in English, play an essential role in both research into and the application of NLP. However, many languages still lack treebanks and building a treebank can be very complicated and difficult. This work has a twofold objective. Firstly, to share our results in constructing a large Vietnamese treebank (VTB) with three levels of annotation including word segmentation, part-of-speech tagging, and syntactic analysis. Major steps in the treebank construction process are described with particular regard to specific Vietnamese properties such as lack of word delimiter and isolation. Those properties make sentences highly syntactically ambiguous, and therefore it is difficult to ensure a high level of agreement among annotators. Various studies of Vietnamese syntax were employed not only to define annotations but also to systematically deal with ambiguities. Annotators were supported by automatic labelling tools, which are based on statistical machine learning methods, for sentence pre-processing and a tree editor for supporting manual annotation. As a result, an annotation agreement of around 90 % was achieved. Our second objective is to present our method for automatically finding errors and inconsistencies in treebank corpora and its application to the construction of the VTB. This method employs the Shannon entropy measure in a manner that the more reduced entropy the more corrected errors in a treebank. The method ranks error candidates by using a scoring function based on conditional entropy. Our experiments showed that this method detected high-error-density subsets of original error candidate sets, and that the corpus entropy was significantly reduced after error correction. The size of these subsets was only about one third of the whole set, while these subsets contained 80-90 % of the total errors. This method can also be applied to languages similar to Vietnamese.

show abstract

Section: Toolsmentioning

confidence: 99%

Vietnamese treebank construction and entropy-based error detection

Nguyen

et al. 2015

Lang Resources & Evaluation

View full text Add to dashboard Cite

show abstract

“…To segment a Vietnamese sentence into clauses, the system first tokens the sentences by VnTagger [6]. Then it segments Vietnamese sentences into clauses using conjunctions and segmentation rules.…”

Section: [Chị Làm Bài Tập Văn] [Em Làm Bài Tập Toán] [Sister Does Litmentioning

confidence: 99%

Sentence Splitting for Vietnamese-English Machine Translation

Hùng

Minh

Shimazu

2012

2012 Fourth International Conference on Knowledge and Systems Engineering

View full text Add to dashboard Cite

Translation quality is often disappointed when a phrase based machine translation system deals with long sentences. Because of syntactic structure discrepancy between two languages, the translation output will not preserve the same word order as the source. When a sentence is long, it should be partitioned into several clauses and the word reordering in the translation should be done within clauses, not between clauses. In this paper, a rule-based technique is proposed to split long Vietnamese sentences based on linguistic information. We use splitting boundaries for translating sentences with two type of constrains: wall and zone. This method is useful for preserving word order and improving translation quality. We describe experiments on translation from Vietnamese to English, showing an improvement BLEU and NIST score.

show abstract

“…In order to perform word segmentation and POS tagging for normalized tweets, we employ vnTokenizer 3 of [10] for word segmentation and VnTagger 4 of [11] for POS tagging.…”

Section: Word Segmentation and Pos Taggingmentioning

confidence: 99%

Named Entity Recognition in Vietnamese Tweets

Nguyen

Snášel

2015

Computational Social Networks

View full text Add to dashboard Cite

Abstract. Named entity recognition (NER) is a task of detecting named entities in documents and categorizing them to predefined classes such as Person (PER), Location (LOC), Organization (ORG) and so on. There have been many approaches proposed to tackle this problem in both formal texts such as news or authorized web content and short texts such as contents in online social network. However, those texts were written in languages other than Vietnamese. In this paper, we propose a method for NER in Vietnamese tweets. Since tweets on Twitter are noisy, irregular, short and consist of acronyms, spelling errors, NER in those tweets is a challenging task. Our method firstly normalizes tweets and then applies a learning model to recognize named entities using six different types of features. We built a training set of more than 40,000 named entities, and a testing set of 2,446 named entities to evaluate our system. The experiment results show that our system achieves encouraging performance with 82.3% F1 score.

show abstract

A Hybrid Approach to Word Segmentation of Vietnamese Texts

Cited by 92 publications

References 5 publications

Vietnamese treebank construction and entropy-based error detection

Vietnamese treebank construction and entropy-based error detection

Sentence Splitting for Vietnamese-English Machine Translation

Named Entity Recognition in Vietnamese Tweets

Contact Info

Product

Resources

About