Lexical Complexity Prediction: An Overview

Abstract

The occurrence of unknown words in texts significantly hinders reading comprehension. To improve accessibility for specific target populations, computational modelling has been applied to identify complex words in texts and substitute them for simpler alternatives. In this paper, we present an overview of computational approaches to lexical complexity prediction focusing on the work carried out on English data. We survey relevant approaches to this problem which include traditional machine learning classifiers (e.g. SVMs, logistic regression) and deep neural networks as well as a variety of features, such as those inspired by literature in psycholinguistics as well as word frequency, word length, and many others. Furthermore, we introduce readers to past competitions and available datasets created on this topic. Finally, we include brief sections on applications of lexical complexity prediction, such as readability and text simplification, together with related studies on languages other than English.

The Integration of Kansei Engineering and Kano Model on Natural Language Processing (NLP) to Support Development of Service Product in the Borobudur Temple Tourism

“…Lexiconbased sentiment analysis is a method of using a dictionary that integrates the polarity of words to determine sentiment [47]. Furthermore, lexicon-based is capable of extracting opinion sentences with very high precision, conveying the language's permanent semantic features as well as the collocational properties of certain text [31].…”

“…Studies using the lexicon and NLP methods have been used to screen for symptoms of depression among Twitter users, yielding better accuracy compared to machine learning [30]. The term "lexicon" refers to a component of NLP system that holds semantic and grammatical information about individual words or strings [31].…”

Features of lexical complexity: insights from L1 and L2 speakers

“…It is well-documented that L2 English speakers make different types of spelling errors compared to L1 English speakers (Napoles et al, 2019 ). However, the connection between spelling error and lexical complexity as defined within the field of natural language processing has been left fairly unexplored (North et al, 2022 ). A doctoral thesis by Wu ( 2013 ) looked into the relationship between self-reported word frequency, familiarity, and morphological complexity with spelling error.…”

PolyU-CBS at TSAR-2022 Shared Task: A Simple, Rank-Based Method for Complex Word Substitution in Two Steps

“…Specifically, in the field of natural language processing (NLP), complexity has often been associated with the difficulties that language users encounter while processing concrete linguistic productions (e.g., sentences, utterances, etc.) (Blache, 2011;Chersoni et al, 2016Chersoni et al, , 2017Chersoni et al, , 2021Sarti et al, 2021;Iavarone et al, 2021), with research focusing on applications that aim to simplify challenging texts and to make them more easily readable for a wider variety of users (North et al, 2022b).…”