Two methods provide similar signals for the need to update systematic reviews

Abstract

Objective Apply and compare two methods that identify signals for the need to update systematic reviews, using three Evidence-based Practice Center reports on omega-3 fatty acids as test cases. Study Design and Setting We applied the RAND method, which uses domain (subject matter) expert guidance, and a modified Ottawa method, which uses quantitative and qualitative signals. For both methods, we conducted focused electronic literature searches of recent studies using the key terms from the original reports. We assessed the agreement between the methods and qualitatively assessed the merits of each system. Results Agreement between the two methods was “substantial” or better (kappa > 0.62) in three of the four systematic reviews. Overall agreement between the methods was “substantial” (kappa = 0.64, 95% confidence interval [CI] 0.45–0.83). Conclusion The RAND and modified Ottawa methods appear to provide similar signals for the possible need to update systematic reviews in this pilot study. Future evaluation with a broader range of clinical topics and eventual comparisons between signals to update reports and the results of full evidence review updates will be needed. We propose a hybrid approach combining the best features of both methods, which should allow efficient review and assessment of the need to update.

Twelve recommendations for integrating existing systematic reviews into new reviews: EPC guidance

“…[13][14][15] We suggest that review authors consider SOE domains approach in synthesis. Using SOE domains as a framework, authors would consider if any new primary studies identified would change the judgments about SOE domains (i.e., study limitations, consistency, precision, directness, and reporting bias).…”

“…However, the process for how to determine if there is sufficient evidence to change a prior grade is an open question. As described above, recent EPC work has addressed when to update a review [13][14][15] and an EPC Workgroup is currently seeking to determine the predictive validity of SOE grading. In general, the judgment is whether enough new evidence exists to change the conclusions or confidence in conclusions.…”

Guidelines for Integration of Systematic Reviews using Primary Studies

“…Most KIs noted that the tools to assess the RB of existing systematic reviews were not appropriate to determine whether the RB of individual studies can be used in the current review. [1][2][3][4][11][12][13][14][15] The most important criterion, according to the KIs, is the type of tool used to evaluate the RB and transparency of the study description. Reviews from presumably reliable sources, such as The Cochrane Collaboration (EPC Program), were preferred.…”

“…Reviews from presumably reliable sources, such as The Cochrane Collaboration (EPC Program), were preferred. [1][2][3][4][11][12][13][14][15] The KIs agreed that an existing review should not necessarily use the same tool to evaluate the RB that would be used in the current review, but the evaluation of the former needs to be performed with an accepted and appropriate tool. In addition, the tools to evaluate the RB should be cited in the study methods.…”

“…However, lack of confidence and the need to repeat the assessment of the RB leads to questions about the possibility of the review being redone or used in the current review. [1][2][3][4]11,12,[14][15][16] Minimum Criteria for the Definition of a Good Systematic Review 8,9…”