Neural Machine Translation of Logographic Languages Using Sub-character Level Information

Zhang, Longtu; Komachi, Mamoru

doi:10.48550/arxiv.1809.02694

Cited by 1 publication

(1 citation statement)

References 0 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Plus, CJK languages do not contain white space as obvious word boundaries in the corpus (Moh and Zhang 2012). Researchers have attempted to mitigate these problems by borrowing information from a parallel corpus of another language, commonly in multilingual corpora for translation (Luo, Tinsley, and Lepage 2013;Zhang and Komachi 2018). Thanks to recent advances in neural networks and pretrained models like BERT (Devlin et al 2018), there has been progress in identifying CJK words that span multiple characters (Hiraoka, Shindo, and Matsumoto 2019).…”

Section: Tokenization For Various Natural Languagesmentioning

confidence: 99%

Comparison of text preprocessing methods

Chai

2022

Nat. Lang. Eng.

View full text Add to dashboard Cite

Text preprocessing is not only an essential step to prepare the corpus for modeling but also a key area that directly affects the natural language processing (NLP) application results. For instance, precise tokenization increases the accuracy of part-of-speech (POS) tagging, and retaining multiword expressions improves reasoning and machine translation. The text corpus needs to be appropriately preprocessed before it is ready to serve as the input to computer models. The preprocessing requirements depend on both the nature of the corpus and the NLP application itself, that is, what researchers would like to achieve from analyzing the data. Conventional text preprocessing practices generally suffice, but there exist situations where the text preprocessing needs to be customized for better analysis results. Hence, we discuss the pros and cons of several common text preprocessing methods: removing formatting, tokenization, text normalization, handling punctuation, removing stopwords, stemming and lemmatization, n-gramming, and identifying multiword expressions. Then, we provide examples of text datasets which require special preprocessing and how previous researchers handled the challenge. We expect this article to be a starting guideline on how to select and fine-tune text preprocessing methods.

show abstract

Section: Tokenization For Various Natural Languagesmentioning

confidence: 99%

Comparison of text preprocessing methods

Chai

2022

Nat. Lang. Eng.

View full text Add to dashboard Cite

show abstract

Neural Machine Translation of Logographic Languages Using Sub-character Level Information

Cited by 1 publication

References 0 publications

Comparison of text preprocessing methods

Comparison of text preprocessing methods

Contact Info

Product

Resources

About