Evaluation of the risk of bias in non-randomized studies of interventions (ROBINS-I) and the ‘target experiment’ concept in studies of exposures: Rationale and preliminary instrument development

Morgan, Rebecca L.; Thayer, Kristina A.; Santesso, Nancy; Holloway, Alison C.; Blain, Robyn; Eftim, Sorina E.; Goldstone, Alexandra E.; Ross, Pam; Guyatt, Gordon; Schünemann, Holger J.

doi:10.1016/j.envint.2018.08.018

Cited by 77 publications

(68 citation statements)

References 15 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Tools for assessing human observational studies of exposure effects need further development 1213. A recent review of 62 tools for assessing observational studies of exposures could not recommend a specific tool but provided guidance for selecting one 14.…”

Section: Unsuitability Of Current Approachesmentioning

confidence: 99%

Making nutrition guidelines fit for purpose

Bero

Norris

Lawrence

2019

BMJ

View full text Add to dashboard Cite

show abstract

Section: Unsuitability Of Current Approachesmentioning

confidence: 99%

Making nutrition guidelines fit for purpose

Bero

Norris

Lawrence

2019

BMJ

View full text Add to dashboard Cite

show abstract

“…The ROBINS-I tool guides adjudications on the risk of bias (ROB) of nonrandomized studies (NRS) of interventions (NRSI), by comparing it to a hypothetical target randomized controlled trial (RCT) that the NRSI best emulates (even if the RCT would be unethical or unfeasible) [2,9]. As ROBINS-I was developed to assess ROB in interventional studies ("intentional exposures"), there was no clarity on its usefulness in assessing ROB in NRS of environmental, nutritional, or other exposures ("unintentional exposures") [10]. Unlike ROBINS-I, most previously available instruments neither use signaling questions nor the comparison to an ideal RCT concept [10].…”

Section: Introductionmentioning

confidence: 99%

“…As ROBINS-I was developed to assess ROB in interventional studies ("intentional exposures"), there was no clarity on its usefulness in assessing ROB in NRS of environmental, nutritional, or other exposures ("unintentional exposures") [10]. Unlike ROBINS-I, most previously available instruments neither use signaling questions nor the comparison to an ideal RCT concept [10].…”

Section: Introductionmentioning

confidence: 99%

“…Risk of bias refers to factors that could have systematically biased the study design, interpretation, analysis, presentation, or reporting of the outcome measures. Due to the intensified focus on the field of exposure assessment by the Grading of Recommendations, Assessment, Development and Evaluation (GRADE) Working Group and the need for harmonizing the rating of NRSI and the NRSE in the context of GRADE [10], a collaborative project led by a team of experts at the University of Bristol (UK), McMaster University (Canada), and the Environmental Protection Agency (USA) proposed to create a new instrument "Risk of Bias in Non-randomized Studies of Exposures" (ROBINS-E) [10]. ROBINS-E is currently under development, and as of the end of 2019, there were no official draft versions available for review [13].…”

Section: Introductionmentioning

confidence: 99%

“…ROBINS-E is currently under development, and as of the end of 2019, there were no official draft versions available for review [13]. What is currently available is a preliminary tool developed by the GRADE working group named the "ROB instrument for NRS of exposures" [10,14]; for simplicity we will use the acronym ROB-NRSE to refer to this tool. ROB-NRSE was developed by making significant semantic and conceptual modifications to the ROBINS-I tool.…”

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

Inter-rater reliability and validity of risk of bias instrument for non-randomized studies of exposures: a study protocol

et al. 2020

View full text Add to dashboard Cite

Background: A new tool, "risk of bias (ROB) instrument for non-randomized studies of exposures (ROB-NRSE)," was recently developed. It is important to establish consistency in its application and interpretation across review teams. In addition, it is important to understand if specialized training and guidance will improve the reliability in the results of the assessments. Therefore, the objective of this cross-sectional study is to establish the inter-rater reliability (IRR), inter-consensus reliability (ICR), and concurrent validity of the new ROB-NRSE tool. Furthermore, as this is a relatively new tool, it is important to understand the barriers to using this tool (e.g., time to conduct assessments and reach consensus-evaluator burden). Methods: Reviewers from four participating centers will apprise the ROB of a sample of NRSE publications using ROB-NRSE tool in two stages. For IRR and ICR, two pairs of reviewers will assess the ROB for each NRSE publication. In the first stage, reviewers will assess the ROB without any formal guidance. In the second stage, reviewers will be provided customized training and guidance. At each stage, each pair of reviewers will resolve conflicts and arrive at a consensus. To calculate the IRR and ICR, we will use Gwet's AC 1 statistic. For concurrent validity, reviewers will appraise a sample of NRSE publications using both the Newcastle-Ottawa Scale (NOS) and ROB-NRSE tool. We will analyze the concordance between the two tools for similar domains and for the overall judgments using Kendall's tau coefficient. To measure evaluator burden, we will assess the time taken to apply ROB-NRSE tool (without and with guidance), and the NOS. To assess the impact of customized training and guidance on the evaluator burden, we will use the generalized linear models. We will use Microsoft Excel and SAS 9.4, to manage and analyze study data, respectively. Discussion: The quality of evidence from systematic reviews that include NRSE depends partly on the study-level ROB assessments. The findings of this study will contribute to an improved understanding of ROB-NRSE and how best to use it.

show abstract