Validation of a Semiautomated Natural Language Processing–Based Procedure for Meta-Analysis of Cancer Susceptibility Gene Penetrance

Abstract

PURPOSE Quantifying the risk of cancer associated with pathogenic mutations in germline cancer susceptibility genes—that is, penetrance—enables the personalization of preventive management strategies. Conducting a meta-analysis is the best way to obtain robust risk estimates. We have previously developed a natural language processing (NLP) –based abstract classifier which classifies abstracts as relevant to penetrance, prevalence of mutations, both, or neither. In this work, we evaluate the performance of this NLP-based procedure. MATERIALS AND METHODS We compared the semiautomated NLP-based procedure, which involves automated abstract classification and text mining, followed by human review of identified studies, with the traditional procedure that requires human review of all studies. Ten high-quality gene–cancer penetrance meta-analyses spanning 16 gene–cancer associations were used as the gold standard by which to evaluate the performance of our procedure. For each meta-analysis, we evaluated the number of abstracts that required human review (workload) and the ability to identify the studies that were included by the authors in their quantitative analysis (coverage). RESULTS Compared with the traditional procedure, the semiautomated NLP-based procedure led to a lower workload across all 10 meta-analyses, with an overall 84% reduction (2,774 abstracts v 16,941 abstracts) in the amount of human review required. Overall coverage was 93%—we are able to identify 132 of 142 studies—before reviewing references of identified studies. Reasons for the 10 missed studies included blank and poorly written abstracts. After reviewing references, nine of the previously missed studies were identified and coverage improved to 99% (141 of 142 studies). CONCLUSION We demonstrated that an NLP-based procedure can significantly reduce the review workload without compromising the ability to identify relevant studies. NLP algorithms have promising potential for reducing human efforts in the literature review process.

Predictive article recommendation using natural language processing and machine learning to support evidence updates in domain-specific knowledge graphs

“…These results are similar to a classification study published recently. 6 Real-world system usage over 12 months resulted in an estimated 99% reduction in volume for SME review, comparable to related work 7 but higher than more stringent classifications for systematic reviews, 31–34 and a yield rate (mean ratio of studies ultimately accepted after review) of 41% which is a significant improvement over 2.94% as reported in pure systematic reviews. 35 …”

“… 3 Various approaches have been developed utilizing natural language processing and machine learning to aid in precision medicine literature curation (reviewed in reference 4 ). Methods span article recommendation techniques 5 and the classification of documents for prioritized human review 6 , 7 to automated identification of gene, mutation, and drug entities, and relationships. 8 SME review of the outputs from such automated systems continues to be an important and necessary step for validating and interpreting 9 , 10 evidence and determining updates in databases.…”

Natural language processing to facilitate breast cancer research and management

“…According to our estimates, this approach saved 700 hours of human effort. We also demonstrated that the semiautomated procedure achieved 93% coverage when used alone and 99% coverage after adding the reference review step (ie, review the references of the initially included papers from the semiautomated approach), meaning it was able to identify almost all penetrance papers that were used for the meta‐analyses …”

“…Once we developed NLP models that could classify penetrance and prevalence with good performance, we attempted to incorporate the most effective model (SVM) into a semiautomated meta‐analysis procedure …”

“…Human reviewers only needed to review the abstracts identified by this NLP plus text mining approach, which was expected to significantly decrease the human effort. The rest of the procedure was identical to the traditional approach …”