Automatic identification of recent high impact clinical articles in PubMed to support clinical decision making using time-agnostic features

Abstract

Objectives: Finding recent clinical studies that warrant changes in clinical practice (“high impact” clinical studies) in a timely manner is very challenging. We investigated a machine learning approach to find recent studies with high clinical impact to support clinical decision making and literature surveillance. Methods: To identify recent studies, we developed our classification model using time-agnostic features that are available as soon as an article is indexed in PubMed®, such as journal impact factor, author count, and study sample size. Using a gold standard of 541 high impact treatment studies referenced in 11 disease management guidelines, we tested the following null hypotheses: 1) the high impact classifier with time-agnostic features (HI-TA) performs equivalently to PubMed’s Best Match sort and a MeSH-based Naïve Bayes classifier; and 2) HI-TA performs equivalently to the high impact classifier with both time-agnostic and time-sensitive features (HI-TS) enabled in a previous study. The primary outcome for both hypotheses was mean top 20 precision. Results: The differences in mean top 20 precision between HI-TA and three baselines (PubMed’s Best Match, a MeSH-based Naïve Bayes classifier, and HI-TS) were not statistically significant (12% vs. 3%, p=0.101; 12% vs. 11%, p=0.720; 12% vs. 25%, p=0.094, respectively). Recall of HI-TA was low (7%). Conclusion: HI-TA had equivalent performance to state-of-the-art approaches that depend on time-sensitive features. With the advantage of relying only on time-agnostic features, the proposed approach can be used as an adjunct to help clinicians identify recent high impact clinical studies to support clinical decision-making. However, low recall limits the use of HI-TA for literature surveillance.

Delay and bias in PubMed medical subject heading (MeSH®) indexing of respiratory journals

“…[4] Developers of machine learning algorithms that will utilise the PubMed database have also identified that time-sensitive features, such as MeSH® indexing and study type, have a detrimental effect on the completeness of a literature search. [5] . CC-BY-NC-ND 4.0 International license It is made available under a perpetuity.…”

“…The copyright holder for this this version posted October 4, 2020. ; https://doi.org/10.1101/2020.10.01.20205476 doi: medRxiv preprint 5 We conducted this study to investigate the delay in MeSH® indexing of asthma articles in key respiratory journals. Our primary aim was to determine the length of delay from publication to indexing and whether delays are increasing over time.…”

“…[4] Developers of machine learning algorithms that will utilise the PubMed database have also identified that time-sensitive features, such as MeSH® indexing and study type, have a detrimental effect on the completeness of a literature search. [5] We conducted this study to investigate the delay in MeSH® indexing of asthma articles in key respiratory journals. Our primary aim was to determine the length of delay from publication to indexing and whether delays are increasing over time.…”

Recent advances of HCI in decision-making tasks for optimized clinical workflows and precision medicine

“…Changes in the clinical practice are obtained also relying upon "high impact" clinical studies that can be retrieved from the PubMed repository. In [123], an ML approach to identify high impact clinical studies in PubMed was presented. Aiming at classifying recently published articles, only static features, mainly independent on the time course, were considered (e.g., journal impact factor, authors' number, study sample size).…”

Automatic Identification of High Impact Relevant Articles to Support Clinical Decision Making Using Attention-Based Deep Learning

“…Plenty of research has been dedicated to designing solutions to obtain a better score in the identification of high impact studies in PubMed literature [1][2][3] and on the topic of matching query-document pairs to encourage the document ranking results [3,4]. The recent development in modern medicine compelled medical professionals to look for relevant information in the secondary databases.…”