Inter-rater and test–retest reliability of quality assessments by novice student raters using the Jadad and Newcastle–Ottawa Scales

Abstract

IntroductionQuality assessment of included studies is an important component of systematic reviews.ObjectiveThe authors investigated inter-rater and test–retest reliability for quality assessments conducted by inexperienced student raters.DesignStudent raters received a training session on quality assessment using the Jadad Scale for randomised controlled trials and the Newcastle–Ottawa Scale (NOS) for observational studies. Raters were randomly assigned into five pairs and they each independently rated the quality of 13–20 articles. These articles were drawn from a pool of 78 papers examining cognitive impairment following electroconvulsive therapy to treat major depressive disorder. The articles were randomly distributed to the raters. Two months later, each rater re-assessed the quality of half of their assigned articles.SettingMcMaster Integrative Neuroscience Discovery and Study Program.Participants10 students taking McMaster Integrative Neuroscience Discovery and Study Program courses.Main outcome measuresThe authors measured inter-rater reliability using κ and the intraclass correlation coefficient type 2,1 or ICC(2,1). The authors measured test–retest reliability using ICC(2,1).ResultsInter-rater reliability varied by scale question. For the six-item Jadad Scale, question-specific κs ranged from 0.13 (95% CI −0.11 to 0.37) to 0.56 (95% CI 0.29 to 0.83). The ranges were −0.14 (95% CI −0.28 to 0.00) to 0.39 (95% CI −0.02 to 0.81) for the NOS cohort and −0.20 (95% CI −0.49 to 0.09) to 1.00 (95% CI 1.00 to 1.00) for the NOS case–control. For overall scores on the six-item Jadad Scale, ICC(2,1)s for inter-rater and test–retest reliability (accounting for systematic differences between raters) were 0.32 (95% CI 0.08 to 0.52) and 0.55 (95% CI 0.41 to 0.67), respectively. Corresponding ICC(2,1)s for the NOS cohort were −0.19 (95% CI −0.67 to 0.35) and 0.62 (95% CI 0.25 to 0.83), and for the NOS case–control, the ICC(2,1)s were 0.46 (95% CI −0.13 to 0.92) and 0.83 (95% CI 0.48 to 0.95).ConclusionsInter-rater reliability was generally poor to fair and test–retest reliability was fair to excellent. A pilot rating phase following rater training may be one way to improve agreement.

Alcohol Consumption and the Risk of Type 2 Diabetes: A Systematic Review and Dose-Response Meta-analysis of More Than 1.9 Million Individuals From 38 Observational Studies

“…Alongside the effect of such factors upon the interpretation of results derived from a quality assessment instrument, the Newcastle-Ottawa tool has received particular criticism. These criticisms range from the tool's focus upon the generalizability of a given sample to the general population as opposed to its internal validity (59) to the arbitrary nature of some questions that appear to weaken interrater reliability (60,61). With these limitations in mind, the Newcastle-Ottawa quality assessment tool should be considered only as a rough guide for readers as opposed to a definitive measure of study quality.…”

Training and calibration of interviewers for oral health literacy using the BREALD-30 in epidemiological studies

“…Total score was the sum of the points assigned to each word in the instrument (0 or 1). Inter-and intra-rater agreement rates higher than 0.85 were considered acceptable levels of agreement 11 (according to the literature, kappa coefficient > 0.80 is considered to be near-perfect agreement 12 and ICC > 0.75 is an excellent agreement 13 ). Furthermore, the percentage of agreement for each word of the BREALD-30 was calculated considering all interviewers in relation to the gold standard researcher.…”

“…20 In the present study, the minimum ICC values for intra-rater agreement of 0.990 and inter-rater agreement of 0.975 are excellent. 13 Although the more heterogeneous the individuals' measures the higher the ICC, this characteristic has been seen as an advantage because it may reduce the disagreement in relation to the magnitude of the measure. 21 The intra rater agreement is necessary because of the subjectivity of some evaluation methods and because of the examiner's specific conditions, such as fatigue, which may lead to inconsistent results.…”

Designing and Psychometric Evaluation of Questionnaire of Influencing Factors of Nursing Students’ Clinical Judgment: A Mixed Method Study in an Iranian Context

“…The internal consistency indicated that the items or dimensions of the questionnaire measured similar variables. The ICC indicated a good value (≥ 0.4) (22). The Cronbach's alpha was reported 0.92.…”