Natural Language Processing and Machine Learning Methods to Characterize Unstructured Patient-Reported Outcomes: Validation Study

Abstract

Background Assessing patient-reported outcomes (PROs) through interviews or conversations during clinical encounters provides insightful information about survivorship. Objective This study aims to test the validity of natural language processing (NLP) and machine learning (ML) algorithms in identifying different attributes of pain interference and fatigue symptoms experienced by child and adolescent survivors of cancer versus the judgment by PRO content experts as the gold standard to validate NLP/ML algorithms. Methods This cross-sectional study focused on child and adolescent survivors of cancer, aged 8 to 17 years, and caregivers, from whom 391 meaning units in the pain interference domain and 423 in the fatigue domain were generated for analyses. Data were collected from the After Completion of Therapy Clinic at St. Jude Children’s Research Hospital. Experienced pain interference and fatigue symptoms were reported through in-depth interviews. After verbatim transcription, analyzable sentences (ie, meaning units) were semantically labeled by 2 content experts for each attribute (physical, cognitive, social, or unclassified). Two NLP/ML methods were used to extract and validate the semantic features: bidirectional encoder representations from transformers (BERT) and Word2vec plus one of the ML methods, the support vector machine or extreme gradient boosting. Receiver operating characteristic and precision-recall curves were used to evaluate the accuracy and validity of the NLP/ML methods. Results Compared with Word2vec/support vector machine and Word2vec/extreme gradient boosting, BERT demonstrated higher accuracy in both symptom domains, with 0.931 (95% CI 0.905-0.957) and 0.916 (95% CI 0.887-0.941) for problems with cognitive and social attributes on pain interference, respectively, and 0.929 (95% CI 0.903-0.953) and 0.917 (95% CI 0.891-0.943) for problems with cognitive and social attributes on fatigue, respectively. In addition, BERT yielded superior areas under the receiver operating characteristic curve for cognitive attributes on pain interference and fatigue domains (0.923, 95% CI 0.879-0.997; 0.948, 95% CI 0.922-0.979) and superior areas under the precision-recall curve for cognitive attributes on pain interference and fatigue domains (0.818, 95% CI 0.735-0.917; 0.855, 95% CI 0.791-0.930). Conclusions The BERT method performed better than the other methods. As an alternative to using standard PRO surveys, collecting unstructured PROs via interviews or conversations during clinical encounters and applying NLP/ML methods can facilitate PRO assessment in child and adolescent cancer survivors.

Artificial Intelligence and Machine Learning in Cancer Related Pain: A Systematic Review

“…Lou et al (2022) [39] delved into patient satisfaction in cancer pain consultation, using LR and RF, revealing associations with physician factors but facing limitations in data collection and ML metric reporting. Lu et al (2021) [59] tested ML algorithms on young cancer survivors, with Bidirectional Encoder Representations from Transformers (BERT) showing higher accuracy in identifying pain interference. Moscato et al (2022) [40] explored physiological signals for pain assessment.…”

Natural Language Processing to Extract Meaningful Information from a Corpus of Written Knowledge in Breast Cancer: Transforming Books into Data

“…Using this strategy, the clinical notes, letters to GP, and some other narrative text (including social media, chats with breast care nurses, etc.) can recreate personal profiles that indicate potential outcomes [20][21][22]. In this way, the corpus of knowledge on a clinical condition will not only be derived from clinical trials, observational studies, or even expert opinions but also from so-called "real world data" [23].…”

Promise and Perils of Large Language Models for Cancer Survivorship and Supportive Care

“…Previous work relevant to cancer survivorship has used smaller language models to automate detection of provider-reported adverse events, [22][23][24] symptoms, 25 and patient-reported outcome characterization. 26 Developing these models historically required a large quantity of manually labeled data to fine-tune the model for specific adverse events or symptoms-an expensive, expertise-intensive task. Better performing LLMs may require much less labeled fine-tuning data for this type of information extraction.…”