Use of Machine Learning Algorithms to Predict the Understandability of Health Education Materials: Development and Evaluation Study

Ji, Meng; Liu, Yanmeng; Zhao, Mengdan; Lyu, Ziqing; Zhang, Boren; Luo, Xin; Liu, Yanlin; Zhong, Yin

doi:10.2196/28413

Cited by 7 publications

(7 citation statements)

References 36 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Mobile Information Systems increasing teaching quality. e work of [14,15] attempts to incorporate globally recognized clinical recommendations into machine learning algorithms to help foreign students in Australian institutions evaluate health resources. Inspired from the above, this paper provides a deep learning-based college English education evaluation method for successfully improving the effect of English teaching evaluation.…”

Section: Related Workmentioning

confidence: 99%

Deep Learning for College English Education Evaluation

Shi

2022

Mobile Information Systems

View full text Add to dashboard Cite

With the expanding and growth of the present reform of English education and teaching, the study of English education quality is receiving more and more attention. The focus of enhancing the quality of English education is to enhance the quality of teaching, and English education assessment is a main initiative to improve the quality of English education and teaching. So, the formation and development of the English education quality assessment system are of great importance to education management. This paper is based on an in-depth study of the present condition and characteristics of English education quality evaluation in colleges and combines the characteristics of deep learning. To provide a feasible solution for the English education evaluation model, this paper constructs a deep learning-based English education quality evaluation model and theoretically analyzes and compares the training results of the deep learning algorithm and the algebraic algorithm used in the model, respectively. Through simulation analysis, we can conclude that the English education evaluation based on deep learning suggested in this paper has achieved good evaluation results.

show abstract

Section: Related Workmentioning

confidence: 99%

Deep Learning for College English Education Evaluation

Shi

2022

Mobile Information Systems

View full text Add to dashboard Cite

show abstract

“…The importance of these other criteria is supported by recent machine learning algorithms that predict the accessibility of health information. For example, these studies have shown that text features such as familiarity with the text's vocabulary and cohesion across sentences may also play an important role [11][12][13][14][15][16].…”

Section: Introductionmentioning

confidence: 99%

Multiple Automated Health Literacy Assessments of Written Health Information: Development of the SHeLL (Sydney Health Literacy Lab) Health Literacy Editor v1

Ayre¹,

Bonner²,

Muscat³

et al. 2023

JMIR Form Res

View full text Add to dashboard Cite

Producing health information that people can easily understand is challenging and time-consuming. Existing guidance is often subjective and lacks specificity. With advances in software that reads and analyzes text, there is an opportunity to develop tools that provide objective, specific, and automated guidance on the complexity of health information. This paper outlines the development of the SHeLL (Sydney Health Literacy Lab) Health Literacy Editor, an automated tool to facilitate the implementation of health literacy guidelines for the production of easy-to-read written health information. Target users were any person or organization that develops consumer-facing education materials, with or without prior experience with health literacy concepts. Anticipated users included health professionals, staff, and government and nongovernment agencies. To develop this tool, existing health literacy and relevant writing guidelines were collated. Items amenable to programmable automated assessment were incorporated into the Editor. A set of natural language processing methods were also adapted for use in the SHeLL Editor, though the approach was primarily procedural (rule-based). As a result of this process, the Editor comprises 6 assessments: readability (school grade reading score calculated using the Simple Measure of Gobbledygook (SMOG)), complex language (percentage of the text that contains public health thesaurus entries, words that are uncommon in English, or acronyms), passive voice, text structure (eg, use of long paragraphs), lexical density and diversity, and person-centered language. These are presented as global scores, with additional, more specific feedback flagged in the text itself. Feedback is provided in real-time so that users can iteratively revise and improve the text. The design also includes a “text preparation” mode, which allows users to quickly make adjustments to ensure accurate calculation of readability. A hierarchy of assessments also helps users prioritize the most important feedback. Lastly, the Editor has a function that exports the analysis and revised text. The SHeLL Health Literacy Editor is a new tool that can help improve the quality and safety of written health information. It provides objective, immediate feedback on a range of factors, complementing readability with other less widely used but important objective assessments such as complex and person-centered language. It can be used as a scalable intervention to support the uptake of health literacy guidelines by health services and providers of health information. This early prototype can be further refined by expanding the thesaurus and leveraging new machine learning methods for assessing the complexity of the written text. User-testing with health professionals is needed before evaluating the Editor’s ability to improve the health literacy of written health information and evaluating its implementation into existing Australian health services.

show abstract

“…For a long time, the study of the quality of language for effective health communication and education has focused on the complexity of health and medical educational resources [ 1 - 5 ]. A range of readability assessment tools have been developed to measure the lexical, grammatical, and syntactic features of health and educational resources [ 6 - 9 ].…”

Section: Introductionmentioning

confidence: 99%

“…Natural language processing tools and machine learning algorithms have gained increasing popularity in health informatics. These flexible and versatile computational techniques can achieve high-precision prediction of outcomes based on the data-driven learning and computing of quantifiable features of the study object [ 1 , 7 , 10 , 20 ]. This represents a significant advance from statistics, which requires the presence of both dependent and independent variables to fit their relations into developed statistical models [ 21 ].…”

Section: Introductionmentioning

confidence: 99%

Predicting the Easiness and Complexity of English Health Materials for International Tertiary Students With Linguistically Enhanced Machine Learning Algorithms: Development and Validation Study

Xie¹,

Ji²,

Hao³

et al. 2021

JMIR Med Inform

Self Cite

View full text Add to dashboard Cite

Background There is an increasing body of research on the development of machine learning algorithms in the evaluation of online health educational resources for specific readerships. Machine learning algorithms are known for their lack of interpretability compared with statistics. Given their high predictive precision, improving the interpretability of these algorithms can help increase their applicability and replicability in health educational research and applied linguistics, as well as in the development and review of new health education resources for effective and accessible health education. Objective Our study aimed to develop a linguistically enriched machine learning model to predict binary outcomes of online English health educational resources in terms of their easiness and complexity for international tertiary students. Methods Logistic regression emerged as the best performing algorithm compared with support vector machine (SVM) (linear), SVM (radial basis function), random forest, and extreme gradient boosting on the transformed data set using L2 normalization. We applied recursive feature elimination with SVM to perform automatic feature selection. The automatically selected features (n=67) were then further streamlined through expert review. The finalized feature set of 22 semantic features achieved a similar area under the curve, sensitivity, specificity, and accuracy compared with the initial (n=115) and automatically selected feature sets (n=67). Logistic regression with the linguistically enhanced feature set (n=22) exhibited important stability and robustness on the training data of different sizes (20%, 40%, 60%, and 80%), and showed consistently high performance when compared with the other 4 algorithms (SVM [linear], SVM [radial basis function], random forest, and extreme gradient boosting). Results We identified semantic features (with positive regression coefficients) contributing to the prediction of easy-to-understand online health texts and semantic features (with negative regression coefficients) contributing to the prediction of hard-to-understand health materials for readers with nonnative English backgrounds. Language complexity was explained by lexical difficulty (rarity and medical terminology), verbs typical of medical discourse, and syntactic complexity. Language easiness of online health materials was associated with features such as common speech act verbs, personal pronouns, and familiar reasoning verbs. Successive permutation of features illustrated the interaction between these features and their impact on key performance indicators of the machine learning algorithms. Conclusions The new logistic regression model developed exhibited consistency, scalability, and, more importantly, interpretability based on existing health and linguistic research. It was found that low and high linguistic accessibilities of online health materials were explained by 2 sets of distinct semantic features. This revealed the inherent complexity of effective health communication beyond current readability analyses, which were limited to syntactic complexity and lexical difficulty.

show abstract

Use of Machine Learning Algorithms to Predict the Understandability of Health Education Materials: Development and Evaluation Study

Cited by 7 publications

References 36 publications

Deep Learning for College English Education Evaluation

Deep Learning for College English Education Evaluation

Multiple Automated Health Literacy Assessments of Written Health Information: Development of the SHeLL (Sydney Health Literacy Lab) Health Literacy Editor v1

Predicting the Easiness and Complexity of English Health Materials for International Tertiary Students With Linguistically Enhanced Machine Learning Algorithms: Development and Validation Study

Contact Info

Product

Resources

About