Comparison of text preprocessing methods

Abstract

Text preprocessing is not only an essential step to prepare the corpus for modeling but also a key area that directly affects the natural language processing (NLP) application results. For instance, precise tokenization increases the accuracy of part-of-speech (POS) tagging, and retaining multiword expressions improves reasoning and machine translation. The text corpus needs to be appropriately preprocessed before it is ready to serve as the input to computer models. The preprocessing requirements depend on both the nature of the corpus and the NLP application itself, that is, what researchers would like to achieve from analyzing the data. Conventional text preprocessing practices generally suffice, but there exist situations where the text preprocessing needs to be customized for better analysis results. Hence, we discuss the pros and cons of several common text preprocessing methods: removing formatting, tokenization, text normalization, handling punctuation, removing stopwords, stemming and lemmatization, n-gramming, and identifying multiword expressions. Then, we provide examples of text datasets which require special preprocessing and how previous researchers handled the challenge. We expect this article to be a starting guideline on how to select and fine-tune text preprocessing methods.

Assessing the Effects of Lemmatisation and Spell Checking on Sentiment Analysis of Online Reviews

“…Two approaches to SA are (1) lexicon based and (2) machine learning based methods [4]. We note that appropriate methods for preprocessing depend on the approach used [2].…”

“…The review has not been converted to all lower case yet, and the punctuation has not been removed or adjusted either. Caution is taken when removing punctuation from the corpus [2], as a reviewer's use of punctuation in an emotional context, e.g. repeated exclamation marks "!!!…”

“…Such data holds value to various stakeholders, creating need for efficient ways to scrape and process this data [1]. In using natural language processing (NLP) techniques, effective preprocessing allows improved detection / interpretation, with significant effects on the results [2].…”

“…Most NLP tools are designed for error free text, whilst the majority of natural text in large corpora is not; typically containing non-standard English, spelling mistakes, colloquialisms, emojis, slang etc. As Chai (2022) states: "[T]ext preprocessing is not a one-size-fits-all process. […] [T]here is not a set of comprehensive guidelines that can advise researchers on which text preprocessing practices to apply to a brand-new text corpus."…”

“…[…] [T]here is not a set of comprehensive guidelines that can advise researchers on which text preprocessing practices to apply to a brand-new text corpus." [2] To address this gap, we investigate preprocessing strategies for online reviews. We focus on detecting sentiment in a specific set of app store reviews to test the efficacy of lemmatisation and spell checking.…”