Grading the quality of evidence in complex interventions: a guide for evidence-based practitioners

Abstract

Evidence-based practitioners who want to apply evidence from complex interventions to the care of their patients are often challenged by the difficulty of grading the quality of this evidence. Using the GRADE (Grading of Recommendations, Assessment, Development and Evaluation) approach and an illustrative example, we propose a framework for evaluating the quality of evidence that depends on obtaining feedback from the evidence user (eg, guideline panel) to inform: (1) proper framing of the question, (2) judgements about directness and consistency of evidence and (3) the need for additional contextual and qualitative evidence. Using this framework, different evidence users and based on their needs would consider the same evidence as high, moderate, low or very low.Complex interventions in medicine are defined as interventions that contain several interacting components within the experimental and control interventions, multiple behaviours required by those delivering or receiving the intervention, multiple groups or organisational levels targeted by the intervention, and permit for variable flexibility and tailoring of the intervention.1 An example is care management programmes for type 2 diabetes. In one systematic review, programmes varied across studies in terms of team composition ( physician, nurse, case manager, diabetes educator), delivery method (face-to-face, telephone, online, outpatient, inpatient), intensity and frequency of the intervention, and other factors. 2 Complex interventions are increasingly used in studies of behavioural change, psychotherapy, education, public health, health services, quality improvement, social policy and many other fields. The challengeJust as with any other intervention, evidence-based practitioners who want to apply evidence from complex intervention to the care of their patients need to select the evidence (hopefully through a systematic review process) and appraise it (ie, identify the extent to which this evidence is trustworthy, also called, rating the quality of evidence). The GRADE (Grading of Recommendations, Assessment, Development and Evaluation) approach is a modern method for rating the quality of evidence with good transparency and reliability. 3 A study of Cochrane systematic reviews showed that the outcomes of complex interventions were more likely to be rated as 'very low' quality of evidence compared with those of simple interventions (37.5% vs 9.1%). None of the outcomes of complex intervention reviews were rated as 'high'. 4 We believe these low ratings are inconsistent with the definition of quality of evidence in GRADE, which is a construct that reflects the trustworthiness of evidence, and we attribute this phenomenon to improper framing of the clinical question for which the quality of evidence is being rated. 5 RationaleComplex interventions are inherently heterogeneous; thus, we may be tempted to lower the quality of evidence for heterogeneity (inconsistency across studies). Complex interventions are also likely to have components that a...

A Framework for Conceptualizing Evidence Needs of Health Systems

“…The Program has recently invested in methods development for evaluating complex multicomponent interventions, 3-9 much of which is applicable to the concerns of health systems. Other work has addressed ratings of SOE for complex interventions 60 and for synthesizing evidence when meta-analyses are not possible; 61 both scenarios are common for interventions considered by health systems. Other authors have provided similar arguments suggesting that a broad definition of evidence including observational studies is necessary for external evidence to be used in system level decision making; 62 and attributing the lack of use of evidence by system level decisionmakers to the historical focus of evidence producers on individual clinicians' needs (as opposed to operational, delivery and policy needs).…”

Diabetes Mellitus Management Among Patients with Limited English Proficiency: A Systematic Review and Meta-Analysis

“…Although the studies in this review had high heterogeneity, the metaanalysis showed a significant reduction in HbA 1c and further presented an opportunity to investigate the effectiveness of various elements of these complex interventions in the subgroup analysis. 32 The majority of studies did not report attrition rates by LEP status. In studies with larger proportions of patients with LEP (≥50%), overall attrition rates ranged from 12% to 33%, with the exception of three outliers, two with a 5% rate and one with a 48% attrition rate.…”

Rating the quality of a body of evidence on the effectiveness of health and social interventions: A systematic review and mapping of evidence domains

“…Other dimensions of complexity are seen as emanating from system properties, that is to say, long, nonlinear, and dynamic relationships between interventions and outcomes, interactions and interdependencies between different components of interventions, and levels of target . Consideration of complexity may require additional guidance when rating the quality of a body of evidence . Study design is often a key issue, given that RCTs are not feasible or appropriate for many population‐level interventions.…”

“…16 Consideration of complexity may require additional guidance when rating the quality of a body of evidence. 11,12,17 Study design is often a key issue, given that RCTs are not feasible or appropriate for many population-level interventions. In addition, many researchers acknowledge that multifaceted heterogeneity between studies in systematic reviews of complex interventions is a more difficult type of problem and requires specific procedures of planning and analysis.…”