Quality assessment tools for evidence from environmental science

Abstract

Assessment of the quality of studies is a critical component of evidence syntheses such as systematic reviews (SRs) that are used to inform policy decisions. To reduce the potential for reviewer bias, and to ensure that the findings of SRs are transparent and reproducible, organisations such as the Cochrane Collaboration, the Campbell Collaboration, and the Collaboration for Environmental Evidence, recommend the use of formal quality assessment tools as opposed to informal expert judgment. However, there is a bewildering array of around 300 formal quality assessment tools that have been identified in the literature, and it has been demonstrated that the use of different tools for the assessment of the same studies can result in different estimates of quality, which can potentially reverse the conclusions of a SR. It is therefore important to consider carefully, the choice of quality assessment tool. We argue that quality assessment tools should: (1) have proven construct validity (i.e. the assessment criteria have demonstrable link with what they purport to measure), (2) facilitate inter-reviewer agreement, (3) be applicable across study designs, and (4) be quick and easy to use. Our aim was to examine current best practice for quality assessment in healthcare and investigate the extent to which these best practices could be useful for assessing the quality of environmental science studies. The feasibility of this transfer is demonstrated in a number of existing SRs on environmental topics. We propose that environmental practitioners should revise, test and adopt the best practice quality assessment tools used in healthcare as a recommended approach for application to environmental science. We provide pilot versions of quality assessment tools, modified from the best practice tools used in healthcare, for application on studies from environmental science.

Can we save large carnivores without losing large carnivore science?

“…Syntheses and literature reviews of large carnivores' ecological roles should identify these issues, but they usually do not, instead routinely failing to assess the internal validity of the original studies reviewed, as described by Bilotta et al (2014). When the individual empirical studies that form the content of these reviews are judged against Platt's (1964) criteria for strong inference, Hone's (2007) deconstruction of experimental design capabilities, or Sutherland et al' (2013) 20 tips for interpreting scientific claims, it is clear that even literature reviews (e.g.…”

What are the consequences of fish entrainment and impingement associated with hydroelectric dams on fish productivity? A systematic review protocol

“…Bilotta et al [28] have outlined criteria for the assessment of the internal validity of a study. Their assessment criteria have been adapted from the Cochrane Collaboration's Risk of Bias Tool [29] for use in the field of environmental science.…”

The effectiveness of spawning habitat creation or enhancement for substrate spawning temperate fish: a systematic review protocol

“…Bilotta et al [24] have outlined criteria for the assessment of the internal validity of a study. Their assessment criteria have been adapted from the Cochrane Collaboration's Risk of Bias Tool [25] for use in the field of environmental science.…”

“…duration of monitoring, endpoints). The criteria outlined in Bilotta et al [24] will be used by the Review Team for this review and included in the reference database. The information for each article retrieved using the search strategy will be uniquely coded based on the criteria (generally categorised as "low risk", high risk", or "unclear risk") to help assess the quality of each study, and to provide insight into any potential risk of bias present in each of the studies.…”