Is unreliability in peer review harmful?

Roediger, Henry L.

doi:10.1017/s0140525x00065936

Cited by 7 publications

(2 citation statements)

References 254 publications

(250 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…For instance, some ascribe unreliability in journal reviews to the purported tendency of editors to select reviewers of differing expertise (Bailar,199 1 ;Kraemer,199 1) or opposite biases about controversial issues (Hargens, 1991;Kiesler, 1991;Marsh & Ball, 1991;Roediger, 1991;Stricker, 1991). For instance, some ascribe unreliability in journal reviews to the purported tendency of editors to select reviewers of differing expertise (Bailar,199 1 ;Kraemer,199 1) or opposite biases about controversial issues (Hargens, 1991;Kiesler, 1991;Marsh & Ball, 1991;Roediger, 1991;Stricker, 1991).…”

Section: Teaching Of Psychologymentioning

confidence: 99%

Peer Review of Term Papers in Graduate Psychology Courses

Haaga

1993

Teaching of Psychology

View full text Add to dashboard Cite

I describe and evaluate a n exercise in which graduate students provide blind reviews of each other's term papers. Besides helping students with their own papers, the exercise seems to help them learn to give constructive, substantive feedback to colleagues. Student ratings of the educational value of peer review are high, but a hoped-for effect of increasing interest in publishing papers has not emerged. A study of reviews from three courses suggested

show abstract

Section: Teaching Of Psychologymentioning

confidence: 99%

Peer Review of Term Papers in Graduate Psychology Courses

Haaga

1993

Teaching of Psychology

View full text Add to dashboard Cite

show abstract

“…Some have described it as "[perhaps] the most important weakness of the peer review process" (Marsh, Bond, & Jayasinghe, 2007, p. 33). Others see low IRR as a fact, neither good nor bad (Roediger, 1991). Others still see low IRR as a desirable feature of peer review (Bailar, 1991;Harnad, 1979;Langfeldt, 2001) as peer reviewers are selected for their diversity and complementary expertise, and it is expected that they disagree.…”

Section: Introductionmentioning

confidence: 99%

Peer reviewer topic choice and its impact on interrater reliability: A mixed-method study

Feliciani

Luo

Shankar

2022

Quantitative Science Studies

View full text Add to dashboard Cite

One of the main critiques of academic peer review is that inter-rater reliability (IRR) among reviewers is low. We examine an under-investigated factor possibly contributing to low IRR, reviewers’ diversity in their topic-criteria mapping (TC-mapping for short). It refers to differences among reviewers pertaining to which topics they choose to emphasize in their evaluations, and how they map those topics onto various evaluation criteria. In this paper we look at the review process of grant proposals in one funding agency to ask: how much do reviewers differ in TC-mapping, and do their differences contribute to low IRR? Through a content analysis of review forms submitted to a national funding agency (Science Foundation Ireland) and a survey of its reviewers, we find evidence of inter-reviewer differences in their TC-mapping. Using a simulation experiment we show that, under a wide range of conditions, even strong differences in TC-mapping only have a negligible impact on IRR. Although further empirical work is needed to corroborate simulation results, these tentatively suggest that reviewers’ heterogeneous TC-mappings might not be of concern for designers of peer review panels to safeguard inter-rater reliability. Peer Review https://publons.com/publon/10.1162/qss_a_00207

show abstract

Peer review reliability: The hierarchy of the sciences

Crothers¹

1993

Behav Brain Sci

View full text Add to dashboard Cite

Is unreliability in peer review harmful?

Cited by 7 publications

References 254 publications

Peer Review of Term Papers in Graduate Psychology Courses

Peer Review of Term Papers in Graduate Psychology Courses

Peer reviewer topic choice and its impact on interrater reliability: A mixed-method study

Peer review reliability: The hierarchy of the sciences

Contact Info

Product

Resources

About