Interrater Reliability in Toxicity Identification: Limitations of Current Standards

Fairchild, A.; Tanksley, Jarred; Tenenbaum, Jessica D.; Palta, Manisha; Hong, Julian C.

doi:10.1016/j.ijrobp.2020.04.040

Cited by 13 publications

(17 citation statements)

References 8 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…As previously described, 100 notes written by 15 physicians were evaluated, representing diverse disease sites ( Table 1 ). 2 No notes were from the same patient or treatment course. Among the most commonly present terms on human review, such as radiation dermatitis, fatigue, nausea, pruritis, and noninfectious cystitis, NLP demonstrated overall good precision, recall, and F1 ( Table 1 ).…”

Section: Resultsmentioning

confidence: 99%

“…We developed an NLP pipeline based on publicly available tools for extracting CTCAE v5.0 terms from oncology notes. As previously described, 2 100 randomly selected notes for weekly scheduled radiotherapy on-treatment visits (OTV) at a single academic center between 2005 and 2016 were independently reviewed by two senior radiation oncology residents.…”

Section: Methodsmentioning

confidence: 99%

“…cough and productive cough). 2 , 14 Labels were then reviewed by an attending radiation oncologist to create a consensus.…”

Section: Methodsmentioning

confidence: 99%

“… 1 However, manual review, whether by a clinician or clinical research assistant, is labor-intensive and prone to human variation. 2 , 3 …”

Section: Introductionmentioning

confidence: 99%

“…Given the limitations and variability in human expert review, 2 the objective of this study was to develop and evaluate an NLP pipeline against human expert reviewers for the extraction of the National Cancer Institute (NCI) Common Terminology Criteria for Adverse Events (CTCAE) symptoms.…”

Section: Introductionmentioning

confidence: 99%

See 4 more Smart Citations

Natural language processing for abstraction of cancer treatment toxicities: accuracy versus human experts

et al. 2020

Self Cite

View full text Add to dashboard Cite

Objectives Expert abstraction of acute toxicities is critical in oncology research but is labor-intensive and variable. We assessed the accuracy of a natural language processing (NLP) pipeline to extract symptoms from clinical notes compared to physicians. Materials and Methods Two independent reviewers identified present and negated National Cancer Institute Common Terminology Criteria for Adverse Events (CTCAE) v5.0 symptoms from 100 randomly selected notes for on-treatment visits during radiation therapy with adjudication by a third reviewer. A NLP pipeline based on Apache clinical Text Analysis Knowledge Extraction System was developed and used to extract CTCAE terms. Accuracy was assessed by precision, recall, and F1. Results The NLP pipeline demonstrated high accuracy for common physician-abstracted symptoms, such as radiation dermatitis (F1 0.88), fatigue (0.85), and nausea (0.88). NLP had poor sensitivity for negated symptoms. Conclusion NLP accurately detects a subset of documented present CTCAE symptoms, though is limited for negated symptoms. It may facilitate strategies to more consistently identify toxicities during cancer therapy.

show abstract

Section: Resultsmentioning

confidence: 99%

Section: Methodsmentioning

confidence: 99%

“…cough and productive cough). 2 , 14 Labels were then reviewed by an attending radiation oncologist to create a consensus.…”

Section: Methodsmentioning

confidence: 99%

“… 1 However, manual review, whether by a clinician or clinical research assistant, is labor-intensive and prone to human variation. 2 , 3 …”

Section: Introductionmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 3 more Smart Citations

Natural language processing for abstraction of cancer treatment toxicities: accuracy versus human experts

et al. 2020

Self Cite

View full text Add to dashboard Cite

show abstract

Whole brain radiotherapy combined with intrathecal liposomal cytarabine for leptomeningeal metastasis—a safety analysis and validation of the EANO-ESMO classification

et al. 2022

View full text Add to dashboard Cite

Background Although there is no proven standard therapy for leptomeningeal metastases (LM), treatment often includes intrathecal chemotherapy combined with whole brain radiation therapy (WBRT). Little is known about the toxicity of such combination therapies. We performed a retrospective safety analysis for the combination of intrathecal liposomal cytarabine with WBRT in patients with LM and validated the EANO-ESMO (European Association of Neuro-oncology—European Society for Medical Oncology) classification in this unique cohort. Methods Treatment toxicities in patients diagnosed with LM between 2004 and 2014 were retrospectively analyzed according to RTOG (Radiation Therapy Oncology Group) toxicity criteria and NCI CTCAE V5.0 (National Cancer Institute Common Terminology Criteria for Adverse Events, Version 5.0). Diagnostic criteria and treatment response as assessed by EANO-ESMO classification were correlated with survival by Kaplan–Meier analysis and Breslow test. Results In all, 40 patients with LM who were treated with combined WBRT and intrathecal cytarabine, were identified. Ten patients (25%) experienced adverse events ≥grade 3 according to RTOG toxicity criteria; in 22 patients (55%) NCI CTCAE ≥grade 3 were detected. Median overall survival was 124 days. Median time to neurological progression was 52 days. Patients with positive cerebrospinal fluid (CSF) cytology (n = 26) showed worse prognosis compared to patients with negative CSF cytology (n = 14; mOS (median overall survival) 84 days versus 198 days, p = 0.006, respectively). The EANO-ESMO response assessment was significantly associated with survival: “stable” (n = 7) mOS 233 days, “response” (n = 10) mOS 206 days, “progression” (n = 17) mOS 45 days, “suspicion of progression” (n = 6) mOS 133 days; overall, p < 0.001. Conclusions In this retrospective analysis, combined treatment of WBRT and intrathecal liposomal cytarabine shows an acceptable safety profile and may indicate a trend towards improved efficacy. The EANO-ESMO classification for diagnosis and treatment response predicts survival.

show abstract

Analysis of Patient-Reported Outcome Utilization Within National Clinical Trials Network Cooperative Group Radiation Oncology Trials Over the Past 2 Decades

Howell

Anker

Walker

et al. 2021

International Journal of Radiation Oncology*Biology*Physics

View full text Add to dashboard Cite

Interrater Reliability in Toxicity Identification: Limitations of Current Standards

Cited by 13 publications

References 8 publications

Natural language processing for abstraction of cancer treatment toxicities: accuracy versus human experts

Natural language processing for abstraction of cancer treatment toxicities: accuracy versus human experts

Whole brain radiotherapy combined with intrathecal liposomal cytarabine for leptomeningeal metastasis—a safety analysis and validation of the EANO-ESMO classification

Analysis of Patient-Reported Outcome Utilization Within National Clinical Trials Network Cooperative Group Radiation Oncology Trials Over the Past 2 Decades

Contact Info

Product

Resources

About