Inter-rater reliability and validity of risk of bias instrument for non-randomized studies of exposures: a study protocol

Abstract

Background: A new tool, "risk of bias (ROB) instrument for non-randomized studies of exposures (ROB-NRSE)," was recently developed. It is important to establish consistency in its application and interpretation across review teams. In addition, it is important to understand if specialized training and guidance will improve the reliability in the results of the assessments. Therefore, the objective of this cross-sectional study is to establish the inter-rater reliability (IRR), inter-consensus reliability (ICR), and concurrent validity of the new ROB-NRSE tool. Furthermore, as this is a relatively new tool, it is important to understand the barriers to using this tool (e.g., time to conduct assessments and reach consensus-evaluator burden). Methods: Reviewers from four participating centers will apprise the ROB of a sample of NRSE publications using ROB-NRSE tool in two stages. For IRR and ICR, two pairs of reviewers will assess the ROB for each NRSE publication. In the first stage, reviewers will assess the ROB without any formal guidance. In the second stage, reviewers will be provided customized training and guidance. At each stage, each pair of reviewers will resolve conflicts and arrive at a consensus. To calculate the IRR and ICR, we will use Gwet's AC 1 statistic. For concurrent validity, reviewers will appraise a sample of NRSE publications using both the Newcastle-Ottawa Scale (NOS) and ROB-NRSE tool. We will analyze the concordance between the two tools for similar domains and for the overall judgments using Kendall's tau coefficient. To measure evaluator burden, we will assess the time taken to apply ROB-NRSE tool (without and with guidance), and the NOS. To assess the impact of customized training and guidance on the evaluator burden, we will use the generalized linear models. We will use Microsoft Excel and SAS 9.4, to manage and analyze study data, respectively. Discussion: The quality of evidence from systematic reviews that include NRSE depends partly on the study-level ROB assessments. The findings of this study will contribute to an improved understanding of ROB-NRSE and how best to use it.

Association between abdominal obesity and diabetic retinopathy in patients with diabetes mellitus: A systematic review and meta-analysis

“…The criteria consists of 11 items, each of which has an answer of either “yes,” “no,” or “unclear”. The quality of case-control and cohort studies was assessed using the Newcastle-Ottawa Scale (NOS), which contains eight items in three major sections: population selection, comparability, and exposure [ 13 ].…”

Exploring school counselors’ and interns’ socioeconomic construct understanding and awareness

“…She found the level of agreement to be 0.80, which is above the recommended inter‐rater reliability (IRR) of 0.75 (Archies et al., 2019). IRR is a validity measure which assesses the consistency of decisions between two reviewers (Jeyaraman et al, 2020).…”

The Reliability Analysis of Speaking Test in Computer-Assisted Language Learning (CALL) Environment

“…Traditionally, during the testing protocol, two reliability components are considered: interrater and intrarater reliability. Jeyaraman et al [58] asserted that interrater reliability refers to the precision of grades provided by evaluators.…”