IntroductionQuality assessment of included studies is an important component of systematic
reviews.ObjectiveThe authors investigated inter-rater and test–retest reliability for
quality assessments conducted by inexperienced student raters.DesignStudent raters received a training session on quality assessment using the Jadad
Scale for randomised controlled trials and the Newcastle–Ottawa Scale (NOS)
for observational studies. Raters were randomly assigned into five pairs and they
each independently rated the quality of 13–20 articles. These articles were
drawn from a pool of 78 papers examining cognitive impairment following
electroconvulsive therapy to treat major depressive disorder. The articles were
randomly distributed to the raters. Two months later, each rater re-assessed the
quality of half of their assigned articles.SettingMcMaster Integrative Neuroscience Discovery and Study Program.Participants10 students taking McMaster Integrative Neuroscience Discovery and Study Program
courses.Main outcome measuresThe authors measured inter-rater reliability using κ and the intraclass
correlation coefficient type 2,1 or ICC(2,1). The authors measured
test–retest reliability using ICC(2,1).ResultsInter-rater reliability varied by scale question. For the six-item Jadad Scale,
question-specific κs ranged from 0.13 (95% CI −0.11 to 0.37) to 0.56
(95% CI 0.29 to 0.83). The ranges were −0.14 (95% CI −0.28 to 0.00)
to 0.39 (95% CI −0.02 to 0.81) for the NOS cohort and −0.20 (95% CI
−0.49 to 0.09) to 1.00 (95% CI 1.00 to 1.00) for the NOS
case–control. For overall scores on the six-item Jadad Scale, ICC(2,1)s for
inter-rater and test–retest reliability (accounting for systematic
differences between raters) were 0.32 (95% CI 0.08 to 0.52) and 0.55 (95% CI 0.41
to 0.67), respectively. Corresponding ICC(2,1)s for the NOS cohort were
−0.19 (95% CI −0.67 to 0.35) and 0.62 (95% CI 0.25 to 0.83), and for
the NOS case–control, the ICC(2,1)s were 0.46 (95% CI −0.13 to 0.92)
and 0.83 (95% CI 0.48 to 0.95).ConclusionsInter-rater reliability was generally poor to fair and test–retest
reliability was fair to excellent. A pilot rating phase following rater training
may be one way to improve agreement.
Women with PTSD stemming from childhood trauma show changes in ToM abilities particularly those often involved in the interpretation of family interactions. In addition, individuals with PTSD showed slower reaction times during the recognition of complex mental states from emotionally salient facial/eye expressions in comparison with healthy subjects.
scite is a Brooklyn-based organization that helps researchers better discover and understand research articles through Smart Citations–citations that display the context of the citation and describe whether the article provides supporting or contrasting evidence. scite is used by students and researchers from around the world and is funded in part by the National Science Foundation and the National Institute on Drug Abuse of the National Institutes of Health.