Applying Natural Language Processing and Hierarchical Machine Learning Approaches to Text Difficulty Classification

Balyan, Renu; McCarthy, Kathryn S.; McNamara, Danielle S.

doi:10.1007/s40593-020-00201-7

Cited by 38 publications

(26 citation statements)

References 60 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…This research should include evaluating the transportability of our approach to deriving literacy profiles from patients' secure messages to diverse health care delivery settings, the development of provider workflow and/or novel population management approaches when patients with limited health literacy are identified, and the effects of interventions that harness this novel source of information on health-related outcomes. We conclude that applying innovative NLP and machine learning approaches 14,15,[51][52][53][54][55][56]83 Andrew J. Karter https://orcid.org/0000-0001-5527-316X…”

Section: Discussionmentioning

confidence: 99%

Employing computational linguistics techniques to identify limited patient health literacy: Findings from the ECLIPPSE study

Schillinger

Balyan

Crossley

et al. 2020

Health Services Research

Self Cite

View full text Add to dashboard Cite

Objective To develop novel, scalable, and valid literacy profiles for identifying limited health literacy patients by harnessing natural language processing. Data Source With respect to the linguistic content, we analyzed 283 216 secure messages sent by 6941 diabetes patients to physicians within an integrated system's electronic portal. Sociodemographic, clinical, and utilization data were obtained via questionnaire and electronic health records. Study Design Retrospective study used natural language processing and machine learning to generate five unique “Literacy Profiles” by employing various sets of linguistic indices: Flesch‐Kincaid (LP_FK); basic indices of writing complexity, including lexical diversity (LP_LD) and writing quality (LP_WQ); and advanced indices related to syntactic complexity, lexical sophistication, and diversity, modeled from self‐reported (LP_SR), and expert‐rated (LP_Exp) health literacy. We first determined the performance of each literacy profile relative to self‐reported and expert‐rated health literacy to discriminate between high and low health literacy and then assessed Literacy Profiles’ relationships with known correlates of health literacy, such as patient sociodemographics and a range of health‐related outcomes, including ratings of physician communication, medication adherence, diabetes control, comorbidities, and utilization. Principal Findings LP_SR and LP_Exp performed best in discriminating between high and low self‐reported (C‐statistics: 0.86 and 0.58, respectively) and expert‐rated health literacy (C‐statistics: 0.71 and 0.87, respectively) and were significantly associated with educational attainment, race/ethnicity, Consumer Assessment of Provider and Systems (CAHPS) scores, adherence, glycemia, comorbidities, and emergency department visits. Conclusions Since health literacy is a potentially remediable explanatory factor in health care disparities, the development of automated health literacy indicators represents a significant accomplishment with broad clinical and population health applications. Health systems could apply literacy profiles to efficiently determine whether quality of care and outcomes vary by patient health literacy; identify at‐risk populations for targeting tailored health communications and self‐management support interventions; and inform clinicians to promote improvements in individual‐level care.

show abstract

Section: Discussionmentioning

confidence: 99%

Employing computational linguistics techniques to identify limited patient health literacy: Findings from the ECLIPPSE study

Schillinger

Balyan

Crossley

et al. 2020

Health Services Research

Self Cite

View full text Add to dashboard Cite

show abstract

“…The modern methodology for text difficulty estimation is based in most cases on machine learning approaches. Thus, R. Balyan et al [3] showed that applying machine learning methods increased accuracy by more than 10% as compared to classic readability metrics (e.g., Flesch-Kincaid formula). To date, a number of studies confirmed the effectiveness of various machine learning techniques for text difficulty estimation, such as support vector machine (SVM) [36,39], random forest [26], and neural networks [2,7,35].…”

Section: Related Workmentioning

confidence: 99%

A Comparative Study of Feature Types for Age-Based Text Classification

Glazkova¹,

Egorov²,

Glazkov³

2021

Lecture Notes in Computer Science

View full text Add to dashboard Cite

The ability to automatically determine the age audience of a novel provides many opportunities for the development of information retrieval tools. Firstly, developers of book recommendation systems and electronic libraries may be interested in filtering texts by the age of the most likely readers. Further, parents may want to select literature for children. Finally, it will be useful for writers and publishers to determine which features influence whether the texts are suitable for children. In this article, we compare the empirical effectiveness of various types of linguistic features for the task of age-based classification of fiction texts. For this purpose, we collected a text corpus of book previews labeled with one of two categories -children's or adult. We evaluated the following types of features: readability indices, sentiment, lexical, grammatical and general features, and publishing attributes. The results obtained show that the features describing the text at the document level can significantly increase the quality of machine learning models. Keywords: Text classification • Fiction • Corpus • Age audience • Content rating • Text difficulty • RuBERT • Neural network • Natural language processing • Machine learning Supported by the grant of the President of the Russian Federation no. MK-637.2020.9.

show abstract

“…Texts range from seven to 80 sentences in length. Rather than relying on shallow measures of readability, the texts were leveled through comparative judgments made by independent raters (for description, see [93]). The initial set of 172 texts was separated into 12 levels of increasing difficulty.…”

Section: Text Set and Questionsmentioning

confidence: 99%

iSTART StairStepper—Using Comprehension Strategy Training to Game the Test

2021

Self Cite

View full text Add to dashboard Cite

Literacy skills are critical for future success, yet over 60% of high school seniors lack proficient reading skills according to standardized tests. The focus on high stakes, standardized test performance may lead educators to “teach-to-the-test” rather than supporting transferable comprehension strategies that students need. StairStepper can fill this gap by blending necessary test prep and reading comprehension strategy practice in a fun, game-based environment. StairStepper is an adaptive literacy skill training game within Interactive Strategy Training for Active Reading and Thinking (iSTART) intelligent tutoring system. StairStepper is unique in that it models text passages and multiple-choice questions of high-stakes assessments, iteratively supporting skill acquisition through self-explanation prompts and scaffolded, adaptive feedback based on performance and self-explanations. This paper describes an experimental study employing a delayed-treatment control design to evaluate users’ perceptions of the StairStepper game and its influence on reading comprehension scores. Results indicate that participants enjoyed the visual aspects of the game environment, wanted to perform well, and considered the game feedback helpful. Reading comprehension scores of students in the treatment condition did not increase. However, the comprehension scores of the control group decreased. Collectively, these results indicate that the StairStepper game may fill the intended gap in instruction by providing enjoyable practice of essential reading comprehension skills and test preparation, potentially increasing students’ practice persistence while decreasing teacher workload.

show abstract

Applying Natural Language Processing and Hierarchical Machine Learning Approaches to Text Difficulty Classification

Cited by 38 publications

References 60 publications

Employing computational linguistics techniques to identify limited patient health literacy: Findings from the ECLIPPSE study

Employing computational linguistics techniques to identify limited patient health literacy: Findings from the ECLIPPSE study

A Comparative Study of Feature Types for Age-Based Text Classification

iSTART StairStepper—Using Comprehension Strategy Training to Game the Test

Contact Info

Product

Resources

About