Outcomes in systematic reviews of complex interventions never reached “high” GRADE ratings when compared with those of simple interventions

Movsisyan, Ani; Melendez‐Torres, G. J.; Montgomery, Paul

doi:10.1016/j.jclinepi.2016.03.014

Cited by 32 publications

(21 citation statements)

References 73 publications

(197 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…We are concerned that continued conflating of 'performance bias' with 'lack of blinding' could have serious downstream effects [18]. For instance, there is empirical evidence that assessments of 'performance bias' by 'presence of blinding participants and providers' could lead to this class of interventions never being able to reach 'high' Grades of Recommendation, Assessment, Development and Evaluation (GRADE) ratings [19]. This could then lead to weaker recommendations for and less uptake of this class of interventions compared to interventions for which this risk of bias assessment is currently designed (e.g.…”

Section: It Is Time To Develop Appropriate Tools For Assessing Minimamentioning

confidence: 99%

It is time to develop appropriate tools for assessing minimal clinically important differences, performance bias and quality of evidence in reviews of behavioral interventions

et al. 2016

View full text Add to dashboard Cite

trials could account fully for any small effects found in our review.Other forms of performance and detection bias are also important. For example, in a systematic review and metaanalysis of 300 randomized trials, Petrosino et al. [3] looked at the impact of non-independent researchers and found that in those trials where programme developers were also the researchers the mean effect size was 0.47, compared with 0.00 when the evaluation team were external and independent. Petrosino et al. concluded that: 'studies in which evaluators were greatly influential in the design and implementation of treatment report consistently and substantially larger effect sizes than other types of evaluators'. The Cochrane risk of bias approach does not include an assessment of this particular risk of bias, and the extent to which programme evaluators were involved in developing and delivering the intervention is not always clear from studies. Therefore, we cannot rule out the possibility that the effect sizes obtained in the current review may be inflated by a conflict-of-interest bias.The fourth point should be discussed more widely among researchers and policymakers: what sort of effect size is good enough? Our interpretation of the effect sizes we were finding across a series of meta-analyses was that they were very small, and unlikely to be of any meaningful benefit on their own, regardless of any possible but unknown reductions in effect sizes due to bias. For example, in the original review, we estimated that the obtained effect sizes would mean (approximately, on average): a decrease in the number of days per week that alcohol was consumed from 2.7 days to 2.5 days; a decrease in the number of drinks consumed each week from 13 to 11 drinks per week; and a decrease in the 69-point Rutgers Alcohol Problems Index (RAPI) from a score of 8.9 to 8.7. We suggest that these achieved effect sizes may fall short of a minimally important clinical difference (MCID). Grant et al. disagree with our interpretation, and that is alright: users of research evidence should make up their own minds on the right interpretation of the evidence.Perhaps an even more important question for the field is this: even if the effects of motivational interviewing for alcohol misuse prevention in young adults are really small, does that mean that we should forget about motivational interviewing as part of a portfolio of different types of prevention interventions [4]? This would be premature, because it is plausible that the whole is greater than the sum of the parts; that the combined effect of multiple single (but individually weak) interventions is worthwhile. However, this should not be seen as a green light for policymakers and practitioners to ignore evidence from randomized controlled trials of single interventions. Rather, it should serve as a focus for the research community. We need better methods for evaluating the combined effect of interventions within prevention systems, and for selecting well-evidenced interventions to incorporate into prev...

show abstract

Section: It Is Time To Develop Appropriate Tools For Assessing Minimamentioning

confidence: 99%

It is time to develop appropriate tools for assessing minimal clinically important differences, performance bias and quality of evidence in reviews of behavioral interventions

et al. 2016

View full text Add to dashboard Cite

show abstract

“…A study of Cochrane systematic reviews showed that the outcomes of complex interventions were more likely to be rated as ‘very low’ quality of evidence compared with those of simple interventions (37.5% vs 9.1%). None of the outcomes of complex intervention reviews were rated as ‘high’ 4. We believe these low ratings are inconsistent with the definition of quality of evidence in GRADE, which is a construct that reflects the trustworthiness of evidence, and we attribute this phenomenon to improper framing of the clinical question for which the quality of evidence is being rated 5…”

Section: The Challengementioning

confidence: 87%

Grading the quality of evidence in complex interventions: a guide for evidence-based practitioners

et al. 2016

View full text Add to dashboard Cite

Evidence-based practitioners who want to apply evidence from complex interventions to the care of their patients are often challenged by the difficulty of grading the quality of this evidence. Using the GRADE (Grading of Recommendations, Assessment, Development and Evaluation) approach and an illustrative example, we propose a framework for evaluating the quality of evidence that depends on obtaining feedback from the evidence user (eg, guideline panel) to inform: (1) proper framing of the question, (2) judgements about directness and consistency of evidence and (3) the need for additional contextual and qualitative evidence. Using this framework, different evidence users and based on their needs would consider the same evidence as high, moderate, low or very low.Complex interventions in medicine are defined as interventions that contain several interacting components within the experimental and control interventions, multiple behaviours required by those delivering or receiving the intervention, multiple groups or organisational levels targeted by the intervention, and permit for variable flexibility and tailoring of the intervention.1 An example is care management programmes for type 2 diabetes. In one systematic review, programmes varied across studies in terms of team composition ( physician, nurse, case manager, diabetes educator), delivery method (face-to-face, telephone, online, outpatient, inpatient), intensity and frequency of the intervention, and other factors. 2 Complex interventions are increasingly used in studies of behavioural change, psychotherapy, education, public health, health services, quality improvement, social policy and many other fields. The challengeJust as with any other intervention, evidence-based practitioners who want to apply evidence from complex intervention to the care of their patients need to select the evidence (hopefully through a systematic review process) and appraise it (ie, identify the extent to which this evidence is trustworthy, also called, rating the quality of evidence). The GRADE (Grading of Recommendations, Assessment, Development and Evaluation) approach is a modern method for rating the quality of evidence with good transparency and reliability. 3 A study of Cochrane systematic reviews showed that the outcomes of complex interventions were more likely to be rated as 'very low' quality of evidence compared with those of simple interventions (37.5% vs 9.1%). None of the outcomes of complex intervention reviews were rated as 'high'. 4 We believe these low ratings are inconsistent with the definition of quality of evidence in GRADE, which is a construct that reflects the trustworthiness of evidence, and we attribute this phenomenon to improper framing of the clinical question for which the quality of evidence is being rated. 5 RationaleComplex interventions are inherently heterogeneous; thus, we may be tempted to lower the quality of evidence for heterogeneity (inconsistency across studies). Complex interventions are also likely to have components that a...

show abstract

“…A similar study by Movsisyan and colleagues appraised a number of systematic reviews to examine the appropriateness of the Grading of Recommendations Assessment, Development and Evaluation (GRADE) tool for interpreting reviews of complex interventions [15, 16]. The authors found that systematic reviews of complex interventions were more frequently categorised as of “very low” quality than those of more “simple” (e.g.…”

Section: Introductionmentioning

confidence: 99%

“…pharmacological) interventions. No single review of a “complex intervention” was deemed to be of “high” quality under the GRADE criteria, despite the presence of studies considered to be the “best possible evidence” in their respective fields [15, 17]. The authors concluded that GRADE, in its current format, may not be suited to the appraisal of such studies.…”

Section: Introductionmentioning

confidence: 99%

Questioning the application of risk of bias tools in appraising evidence from natural experimental studies: critical reflections on Benton et al., IJBNPA 2016

Humphreys

Panter

Ogilvie

2017

Int J Behav Nutr Phys Act

View full text Add to dashboard Cite

We recently read the article by Benton et al. which reviewed risk of bias in natural experimental studies investigating the impact of the built environment on physical activity (Benton et al., 2016; Int J Behav Nutr Phys Act 13:107). As a technical exercise in assessing risk of bias to understand study quality, we found the results of this study both interesting and potentially useful. However, it prompted a number of concerns with the use of risk of bias tools for assessing the quality of evidence from studies exploiting natural experiments. As we discuss in this commentary, the rigid application of such tools could have adverse effects on the uptake and use of natural experiments in population health research and practice.

show abstract

Outcomes in systematic reviews of complex interventions never reached “high” GRADE ratings when compared with those of simple interventions

Cited by 32 publications

References 73 publications

It is time to develop appropriate tools for assessing minimal clinically important differences, performance bias and quality of evidence in reviews of behavioral interventions

It is time to develop appropriate tools for assessing minimal clinically important differences, performance bias and quality of evidence in reviews of behavioral interventions

Grading the quality of evidence in complex interventions: a guide for evidence-based practitioners

Questioning the application of risk of bias tools in appraising evidence from natural experimental studies: critical reflections on Benton et al., IJBNPA 2016

Contact Info

Product

Resources

About