Word Complexity is in the Eye of the Beholder

Gooding, Sian; Kochmar, Ekaterina; Yimam, Seid Muhie; Biemann, Chris

doi:10.18653/v1/2021.naacl-main.351

Cited by 8 publications

(14 citation statements)

References 29 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In order to facilitate personalised CWI, and therefore personalised text simplification and readability systems, audience specific complexity annotations are required (Bingel et al, 2018). Whilst it has been shown that the concept of word complexity, and thus the level of agreement, is aligned between individuals sharing a common background (Gooding and Kochmar, 2018a;Gooding et al, 2021), we argue that the best CWI model for each individual is trained with them 'in the loop'.…”

Section: Introductionmentioning

confidence: 80%

“…However, annotating the difficulty of words is a subjective task, and previous data collection has yielded low levels of annotator agreement (Specia et al, 2012;Paetzold and Specia, 2016c). As a way of mitigating individual differences, such datasets typically present a homogeneous view on word complexity, by merging annotations across readers (Gooding et al, 2021). Further attempts to improve annotator agreement have included offering bonus incentives for annotators who select words matching other annotations (Yimam et al, 2017), as well as providing guidelines for annotators to mark words that they assume would be complex for audiences such as children and those with learning difficulties.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

One Size Does Not Fit All: The Case for Personalised Word Complexity Models

Gooding¹,

Manuel²

2022

Findings of the Association for Computational Linguistics: NAACL 2022

Self Cite

View full text Add to dashboard Cite

Complex Word Identification (CWI) aims to detect words within a text that a reader may find difficult to understand. It has been shown that CWI systems can improve text simplification, readability prediction and vocabulary acquisition modelling. However, the difficulty of a word is a highly idiosyncratic notion that depends on a reader's first language, proficiency and reading experience. In this paper, we show that personal models are best when predicting word complexity for individual readers. We use a novel active learning framework that allows models to be tailored to individuals and release a dataset of complexity annotations and models as a benchmark for further research.

show abstract

Section: Introductionmentioning

confidence: 80%

Section: Introductionmentioning

confidence: 99%

One Size Does Not Fit All: The Case for Personalised Word Complexity Models

Gooding¹,

Manuel²

2022

Findings of the Association for Computational Linguistics: NAACL 2022

Self Cite

View full text Add to dashboard Cite

show abstract

“…This finding correlates well with existing research on readability. Linguistic features, such as word length, word frequency, and word familiarity, and other structural features have proven to be highly relevant and reliable predictors of textual complexity and difficulty [ 39 ].…”

Section: Discussionmentioning

confidence: 99%

Predicting the Easiness and Complexity of English Health Materials for International Tertiary Students With Linguistically Enhanced Machine Learning Algorithms: Development and Validation Study

Xie¹,

Ji²,

Hao³

et al. 2021

JMIR Med Inform

View full text Add to dashboard Cite

Background There is an increasing body of research on the development of machine learning algorithms in the evaluation of online health educational resources for specific readerships. Machine learning algorithms are known for their lack of interpretability compared with statistics. Given their high predictive precision, improving the interpretability of these algorithms can help increase their applicability and replicability in health educational research and applied linguistics, as well as in the development and review of new health education resources for effective and accessible health education. Objective Our study aimed to develop a linguistically enriched machine learning model to predict binary outcomes of online English health educational resources in terms of their easiness and complexity for international tertiary students. Methods Logistic regression emerged as the best performing algorithm compared with support vector machine (SVM) (linear), SVM (radial basis function), random forest, and extreme gradient boosting on the transformed data set using L2 normalization. We applied recursive feature elimination with SVM to perform automatic feature selection. The automatically selected features (n=67) were then further streamlined through expert review. The finalized feature set of 22 semantic features achieved a similar area under the curve, sensitivity, specificity, and accuracy compared with the initial (n=115) and automatically selected feature sets (n=67). Logistic regression with the linguistically enhanced feature set (n=22) exhibited important stability and robustness on the training data of different sizes (20%, 40%, 60%, and 80%), and showed consistently high performance when compared with the other 4 algorithms (SVM [linear], SVM [radial basis function], random forest, and extreme gradient boosting). Results We identified semantic features (with positive regression coefficients) contributing to the prediction of easy-to-understand online health texts and semantic features (with negative regression coefficients) contributing to the prediction of hard-to-understand health materials for readers with nonnative English backgrounds. Language complexity was explained by lexical difficulty (rarity and medical terminology), verbs typical of medical discourse, and syntactic complexity. Language easiness of online health materials was associated with features such as common speech act verbs, personal pronouns, and familiar reasoning verbs. Successive permutation of features illustrated the interaction between these features and their impact on key performance indicators of the machine learning algorithms. Conclusions The new logistic regression model developed exhibited consistency, scalability, and, more importantly, interpretability based on existing health and linguistic research. It was found that low and high linguistic accessibilities of online health materials were explained by 2 sets of distinct semantic features. This revealed the inherent complexity of effective health communication beyond current readability analyses, which were limited to syntactic complexity and lexical difficulty.

show abstract

“…Text simplification has many subtleties, as what would be a valid simplification for one reader may not be appropriate for another (Xu et al, 2015). For instance, it has been shown that the factors contributing to word complexity vary depending on the first language and proficiency level of a reader (Gooding et al, 2021b). The subjective nature of text simplification means that system evaluation is difficult.…”

Section: Datasets and Evaluationmentioning

confidence: 99%

“…In automatic text simplification, the aim is to transform text using the aforementioned operations, to allow individuals with differing comprehension levels access. This requires a fundamental understanding of what factors contribute to text complexity for differing audiences (Gooding et al, 2021b).…”

Section: Introductionmentioning

confidence: 99%

On the Ethical Considerations of Text Simplification

Gooding¹

2022

Ninth Workshop on Speech and Language Processing for Assistive Technologies (SLPAT-2022)

Self Cite

View full text Add to dashboard Cite

This paper outlines the ethical implications of text simplification within the framework of assistive systems. We argue that a distinction should be made between the technologies that perform text simplification and the realisation of these in assistive technologies. When using the latter as a motivation for research, it is important that the subsequent ethical implications be carefully considered. We provide guidelines for the framing of text simplification independently of assistive systems, as well as suggesting directions for future research and discussion based on the concerns raised.

show abstract

Word Complexity is in the Eye of the Beholder

Cited by 8 publications

References 29 publications

One Size Does Not Fit All: The Case for Personalised Word Complexity Models

One Size Does Not Fit All: The Case for Personalised Word Complexity Models

Predicting the Easiness and Complexity of English Health Materials for International Tertiary Students With Linguistically Enhanced Machine Learning Algorithms: Development and Validation Study

On the Ethical Considerations of Text Simplification

Contact Info

Product

Resources

About