Are the confidence scores of reviewers consistent with the review content? Evidence from top conference proceedings in AI

Wu, Wenqing; Xi, Haixu; Zhang, Chengzhi

doi:10.1007/s11192-024-05070-8

Scientometrics

2024

DOI: 10.1007/s11192-024-05070-8

|View full text |Cite

Are the confidence scores of reviewers consistent with the review content? Evidence from top conference proceedings in AI

Wenqing Wu,

Haixu Xi,

Chengzhi Zhang

Help me understand this report

Search citation statements

Order By: Relevance

Paper Sections

Select...

Citation Types

Supporting

Mentioning

Contrasting

Year Published

2024

Publication Types

Select...

Article1

Relationship

Self Cite0

Independent1

Authors

Journals

Cited by 1 publication

References 41 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

Examining uncertainty in journal peer reviewers’ recommendations: a cross-sectional study

Barnett,

Allen,

Aldcroft

et al. 2024

R. Soc. Open Sci.

View full text Add to dashboard Cite

The peer review process is used throughout science but has often been criticized for being inconsistent, with decisions dependent on the peers who did the reviewing. Much of the decision inconsistency arises from the differences between reviewers in terms of their expertise, training and experience. Another source of uncertainty is within reviewers as they must make a single recommendation (e.g. ‘Accept’), when they may have wavered between two (e.g. ‘Accept’ or ‘Reject’). We estimated the size of within-reviewer uncertainty using post-review surveys at three journals. We asked reviewers to think outside the recommendation they gave (e.g. ‘Accept’) and assign percentages to all other recommendations (e.g. ‘Major revision’). Reviewers who were certain could assign 100% to one recommendation. Twenty-three per cent of reviewers reported no uncertainty (95% confidence interval 19–27%). Women were associated with more uncertainty at one journal, and protocol papers were associated with more uncertainty at one journal. Reviewers commonly experience some uncertainty when peer-reviewing journal articles. This uncertainty is part of the variability in peer reviewers’ recommendation.

show abstract

Examining uncertainty in journal peer reviewers’ recommendations: a cross-sectional study

Barnett,

Allen,

Aldcroft

et al. 2024

R. Soc. Open Sci.

View full text Add to dashboard Cite

show abstract

scite is a Brooklyn-based organization that helps researchers better discover and understand research articles through Smart Citations–citations that display the context of the citation and describe whether the article provides supporting or contrasting evidence. scite is used by students and researchers from around the world and is funded in part by the National Science Foundation and the National Institute on Drug Abuse of the National Institutes of Health.

Contact Info

hi@scite.ai

10624 S. Eastern Ave., Ste. A-614

Henderson, NV 89052, USA

Blog Terms and Conditions API Terms Privacy Policy Contact Cookie Preferences Do Not Sell or Share My Personal Information

Made with 💙 for researchers

Part of the Research Solutions Family.

Are the confidence scores of reviewers consistent with the review content? Evidence from top conference proceedings in AI

Cited by 1 publication

References 41 publications

Examining uncertainty in journal peer reviewers’ recommendations: a cross-sectional study

Examining uncertainty in journal peer reviewers’ recommendations: a cross-sectional study

Contact Info

Product

Resources

About