Applying the ROBINS-I tool to natural experiments: an example from public health

Abstract

BackgroundA new tool to assess Risk of Bias In Non-randomised Studies of Interventions (ROBINS-I) was published in Autumn 2016. ROBINS-I uses the Cochrane-approved risk of bias (RoB) approach and focusses on internal validity. As such, ROBINS-I represents an important development for those conducting systematic reviews which include non-randomised studies (NRS), including public health researchers. We aimed to establish the applicability of ROBINS-I using a group of NRS which have evaluated non-clinical public health natural experiments.MethodsFive researchers, all experienced in critical appraisal of non-randomised studies, used ROBINS-I to independently assess risk of bias in five studies which had assessed the health impacts of a domestic energy efficiency intervention. ROBINS-I assessments for each study were entered into a database and checked for consensus across the group. Group discussions were used to identify reasons underpinning lack of consensus for specific questions and bias domains.ResultsROBINS-I helped to systematically articulate sources of bias in NRS. However, the lack of consensus in assessments for all seven bias domains raised questions about ROBINS-I’s reliability and applicability for natural experiment studies. The two RoB domains with least consensus were selection (Domain 2) and performance (Domain 4). Underlying the lack of consensus were difficulties in applying an intention to treat or per protocol effect of interest to the studies. This was linked to difficulties in determining whether the intervention status was classified retrospectively at follow-up, i.e. post hoc. The overall risk of bias ranged from moderate to critical; this was most closely linked to the assessment of confounders.ConclusionThe ROBINS-I tool is a conceptually rigorous tool which focusses on risk of bias due to the counterfactual. Difficulties in applying ROBINS-I may be due to poor design and reporting of evaluations of natural experiments. While the quality of reporting may improve in the future, improved guidance on applying ROBINS-I is needed to enable existing evidence from natural experiments to be assessed appropriately and consistently. We hope future refinements to ROBINS-I will address some of the issues raised here to allow wider use of the tool.

Association between suicide reporting in the media and suicide: systematic review and meta-analysis

“…Studies were considered at low risk of bias if all domains were coded as low risk; at moderate risk if at least one domain was coded moderate but none as serious; at serious risk if at least one domain was assessed as serious but none as critical; and at critical risk if any domain was coded as critical. Like an earlier study that applied the Robins-I tool to natural experiments,30 we found that the first domain, risk of bias as a result of confounding, generally determined the overall risk of bias. This domain comprised coding for subdomains if the number of pre-intervention times was sufficient to allow characterisation of the series; appropriate analysis techniques were used to account for time trends and time patterns; seasonality was accounted for; and possible confounders were measured and controlled for.…”

“…This tool was originally designed for non-randomised cohort studies, and does not directly apply to our study designs. The general concept, however, is applicable to interrupted time series designs,30 and the authors of Robins-I have published on issues that will be looked at in a future version for studies of interrupted time series 31. We developed a specific adaption for this study with six domains of bias: bias as a result of confounding issues; bias in classification of interventions; bias because of preparatory phases; bias because of missing data; bias in measurement of the outcome; and bias in selection of reported results.…”

Individual risk factors predictive of venous thromboembolism in patients with temporary lower limb immobilization due to injury: a systematic review

“…The tool is based on the original Cochrane risk of bias tool for randomized studies and also builds on related tools such as QUADAS‐2 (Quality Assessment of Diagnostic Accuracy Studies) . ROBINS‐I provides a detailed framework for assessment and judgement of risk of bias domains, and has been used previously within the systematic review literature .…”

Neuromuscular taping application opportunities in nursing: a literature review

“…Interventions" (ROBINS-I) tool that focuses on evaluating internal validity through the measurement of seven specific domains: confounding, selection of participants into study, classification of interventions, deviation from intended interventions, missing data, measurement of outcomes, and selection of reported results (Thomson et al, 2018).…”