How good are my search strings? Reflections on using an existing review as a quasi-gold standard

Tran, Huynh Khanh Vi; Börstler, Jürgen; Ali, Nauman bin; Unterkalmsteiner, Michael

doi:10.37190/e-inf220103

Cited by 4 publications

(3 citation statements)

References 30 publications

(139 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Lastly, with the key paper heuristic, a set of important papers is determined beforehand, for example, by expert consensus, and the screener stops if all these papers are found with active learning. This method is often used for validating the search strategy by ensuring that the search process adequately identifies relevant primary studies [8,42].…”

Section: Introductionmentioning

confidence: 99%

The SAFE procedure: a practical stopping heuristic for active learning-based screening in systematic reviews and meta-analyses

Boetje,

van de Schoot

2024

Syst Rev

View full text Add to dashboard Cite

Active learning has become an increasingly popular method for screening large amounts of data in systematic reviews and meta-analyses. The active learning process continually improves its predictions on the remaining unlabeled records, with the goal of identifying all relevant records as early as possible. However, determining the optimal point at which to stop the active learning process is a challenge. The cost of additional labeling of records by the reviewer must be balanced against the cost of erroneous exclusions. This paper introduces the SAFE procedure, a practical and conservative set of stopping heuristics that offers a clear guideline for determining when to end the active learning process in screening software like ASReview. The eclectic mix of stopping heuristics helps to minimize the risk of missing relevant papers in the screening process. The proposed stopping heuristic balances the costs of continued screening with the risk of missing relevant records, providing a practical solution for reviewers to make informed decisions on when to stop screening. Although active learning can significantly enhance the quality and efficiency of screening, this method may be more applicable to certain types of datasets and problems. Ultimately, the decision to stop the active learning process depends on careful consideration of the trade-off between the costs of additional record labeling against the potential errors of the current model for the specific dataset and context.

show abstract

Section: Introductionmentioning

confidence: 99%

The SAFE procedure: a practical stopping heuristic for active learning-based screening in systematic reviews and meta-analyses

Boetje,

van de Schoot

2024

Syst Rev

View full text Add to dashboard Cite

show abstract

“…A variation of the number-based approach is to screen a predefined set of records randomly and use the observed fraction of relevant records to extrapolate an estimate of relevant records for the complete set (van Haastrecht et al, 2021). Lastly, we propose the key paper heuristic to validate the recall, which has been described in various sources, such as Tran et al (2022), andBramer et al (2018). Key papers are typically used for validating the search strategy by ensuring that the search process adequately identifies relevant primary studies.…”

Section: Introductionmentioning

confidence: 99%

The SAFE Procedure: A Practical Stopping Heuristic for Active Learning-Based Screening in Systematic Reviews and Meta-Analyses

Boetje,

van de Schoot

2023

Preprint

View full text Add to dashboard Cite

Active learning has become an increasingly popular method for screening large amounts of data in systematic reviews and meta-analyses. The active learning process continually improves its predictions on the remaining unlabeled records, with the goal of identifying all relevant records as early as possible. However, determining the optimal point at which to stop the active learning process is a challenge. The cost of additional labeling of records by the reviewer must be balanced against the cost of errors made by the current model. This paper introduces the SAFE procedure, a practical and conservative set of stopping heuristics that offers a clear guideline for determining when to end the active learning process in screening software like ASReview. Our main conclusion of this paper is that relying on a single stopping rule is not sufficient and employing an eclectic mix of stopping heuristics helps to minimize the risk of missing relevant papers in the screening processThe SAFE procedure combines different heuristics to avoid stopping too early and potentially missing relevant records. The SAFE procedure takes into account the model's accuracy and uncertainty, as well as the cost of continuing to label records. This procedure for using active learning in systematic literature review screening provides a practical and efficient approach that can save significant time and resources while ensuring a conservative approach to determining when to end the active learning process. The proposed stopping heuristic balances the costs of continued screening with the risk of missing relevant records, providing a practical solution for reviewers to make informed decisions on when to stop screening. The proposed method can assist researchers in identifying relevant records early, which can ultimately lead to improved evidence synthesis and decision-making in many fields.

show abstract

“…Lastly, we propose the key paper heuristic to validate the recall, which has been described in various sources, such as Tran et al (2022), and Bramer et al (2018). Key papers are typically used for validating the search strategy by ensuring that the search process adequately identi es relevant primary studies.…”

Section: Introductionmentioning

confidence: 99%

The SAFE Procedure: A Practical Stopping Heuristic for Active Learning-Based Screening in Systematic Reviews and Meta-Analyses

Boetje,

de Schoot

2023

Preprint

View full text Add to dashboard Cite

Active learning has become an increasingly popular method for screening large amounts of data in systematic reviews and meta-analyses. The active learning process continually improves its predictions on the remaining unlabeled records, with the goal of identifying all relevant records as early as possible. However, determining the optimal point at which to stop the active learning process is a challenge. The cost of additional labeling of records by the reviewer must be balanced against the cost of errors made by the current model. This paper introduces the SAFE procedure, a practical and conservative set of stopping heuristics that offers a clear guideline for determining when to end the active learning process in screening software like ASReview. Our main conclusion of this paper is that relying on a single stopping rule is not sufficient and employing an eclectic mix of stopping heuristics helps to minimize the risk of missing relevant papers in the screening processThe SAFE procedure combines different heuristics to avoid stopping too early and potentially missing relevant records. The SAFE procedure takes into account the model's accuracy and uncertainty, as well as the cost of continuing to label records. This procedure for using active learning in systematic literature review screening provides a practical and efficient approach that can save significant time and resources while ensuring a conservative approach to determining when to end the active learning process. The proposed stopping heuristic balances the costs of continued screening with the risk of missing relevant records, providing a practical solution for reviewers to make informed decisions on when to stop screening. The proposed method can assist researchers in identifying relevant records early, which can ultimately lead to improved evidence synthesis and decision-making in many fields.

show abstract

How good are my search strings? Reflections on using an existing review as a quasi-gold standard

Cited by 4 publications

References 30 publications

The SAFE procedure: a practical stopping heuristic for active learning-based screening in systematic reviews and meta-analyses

The SAFE procedure: a practical stopping heuristic for active learning-based screening in systematic reviews and meta-analyses

The SAFE Procedure: A Practical Stopping Heuristic for Active Learning-Based Screening in Systematic Reviews and Meta-Analyses

The SAFE Procedure: A Practical Stopping Heuristic for Active Learning-Based Screening in Systematic Reviews and Meta-Analyses

Contact Info

Product

Resources

About