Creating enriched training sets of eligible studies for large systematic reviews: the utility of PubMed's Best Match algorithm

Abstract

Introduction Solutions like crowd screening and machine learning can assist systematic reviewers with heavy screening burdens but require training sets containing a mix of eligible and ineligible studies. This study explores using PubMed's Best Match algorithm to create small training sets containing at least five relevant studies. Methods Six systematic reviews were examined retrospectively. MEDLINE searches were converted and run in PubMed. The ranking of included studies was studied under both Best Match and Most Recent sort conditions. Results Retrieval sizes for the systematic reviews ranged from 151 to 5,406 records and the numbers of relevant records ranged from 8 to 763. The median ranking of relevant records was higher in Best Match for all six reviews, when compared with Most Recent sort. Best Match placed a total of thirty relevant records in the first fifty, at least one for each systematic review. Most Recent sorting placed only ten relevant records in the first fifty. Best Match sorting outperformed Most Recent in all cases and placed five or more relevant records in the first fifty in three of six cases. Discussion Using a predetermined set size such as fifty may not provide enough true positives for an effective systematic review training set. However, screening PubMed records ranked by Best Match and continuing until the desired number of true positives are identified is efficient and effective. Conclusions The Best Match sort in PubMed improves the ranking and increases the proportion of relevant records in the first fifty records relative to sorting by recency.

Artificial intelligence behind the scenes: PubMed’s Best Match algorithm

“…We would like to compare search results in Best Match between expert searchers and nonexpert searchers such as students, doctors, and clinical researchers by asking each group to run a search on a query and observing how they search, determine relevance, and select articles to read. Previously, Sampson performed a study in which she used Best Match as a ranker for results from a systematic review search and found that Best Match sort order “placed three times as many relevant records in the top fifty than Most Recent” results [ 29 ]. However, her study used searches designed by expert searchers.…”

Information Retrieval and Knowledge Organization: A Perspective from the Philosophy of Science

“…(It is no longer true that retrieved sets are not ranked. In 2019, "best match" replaced "most recent" as the default sort order for search results in PubMed, cf., [21,22]).…”

New challenges and solutions for information retrieval: information specialists as “bricklayers” in the health technology assessment framework

“…This issue also contains a timely methodological evaluation of new database features -the PubMed "Best Match" sorting option evaluated by Sampson et al (9). This is a good example of the methodological support offered by information specialists: new database features or techniques are evaluated and can then be applied according to the evidence.…”