How stakeholder engagement has led us to reconsider definitions of rigour in systematic reviews

Abstract

As a methodology designed to inform policy and practice decisions, it is particularly important to ensure that systematic reviews are shaped by those who will use them. There is a broad range of approaches for engagement of the potential users of reviews that aim to elicit their priorities and needs and incorporate these into the review design. This incorporation of their priorities and needs can create a tension between their calls for locally-specific, often rapidly-produced evidence syntheses for policy needs and the production of unbiased, generalisable, globally-relevant systematic reviews. This tension raises the question of what is a 'gold standard' review. This commentary aims to address head on this often undiscussed key challenge with regard to stakeholder involvement in systematic reviews: that responding to stakeholders can mean reconsidering what makes a review rigorous. The commentary proposes a new model to address these tensions that combines the production of public-good reviews, with stakeholder-driven syntheses. In this, it presents the approach taken by our team at the Africa Centre for Evidence at the University of Johannesburg to achieve two different but complementary outputs: (i) 'public goods' , namely comprehensive and generalisable systematic reviews of the evidence available for and accessible to a global audience, and (ii) locallyfocussed, stakeholder-driven, pragmatically-produced syntheses for decision-making at a policy level. The designed approach incorporates balancing the formal requirements of full, published systematic reviews with engagement of national and international decision-makers. It also accommodates space to move from stakeholder engagement to co-production, where stakeholders are engaged to such an extent that they become partners in the production of the review. These approaches are integrated into the traditional steps for producing a systematic review with implications as to what constitutes a gold standard approach to synthesising evidence.

What evidence exists for the impact of Baltic Sea ecosystems on human health and well-being? A systematic map protocol

“…From the outset, we have taken a transdisciplinary approach, with plans to involve stakeholders at the different phases of the review process [13,22,23]. The aim is to gain insight into the policy relevance of emerging risks and benefits to public health from ecosystem services supplied by the Baltic Sea and how decision-makers could be supported in developing appropriate evidence-based policies.…”

Using Evidence in Policy and Practice

“…More recent reviews of models of evidence use criticise their strong emphasis on the supply of evidence and abstract academic definitions of use (Newman et al, 2013;Langer et al, 2017). This has led to a more inclusive theory of evidence use in which users can be co-producers of knowledge and evidence rather than mere consumers (e.g.…”

A five-step approach for stakeholder engagement in prioritisation and planning of environmental evidence syntheses

“…[5]), we have experienced that it can sometimes be challenging to reconcile stakeholders' desires and expectations with established methods for evidence synthesis (as discussed in "Identification of policy-and practice-relevant topics"and exemplified by the review on PFASs in "Framing and prioritisation of review questions"). Other authors have argued that a more pragmatic approach, where urgent needs of local stakeholders potentially compromise the comprehensiveness of the review, may be justified [45]. However, the iterative process of prioritisation and scoping employed by EviEM, which involves a continuous dialogue between reviewers, scientists and other stakeholders, usually contributes to scientific rigour while retaining the relevance of reviews to stakeholders.…”