On peer review in computer science: analysis of its effectiveness and suggestions for improvement

Ragone, Azzurra; Мирыленка, Кацярына; Casati, Fabio; Marchese, Maurizio

doi:10.1007/s11192-013-1002-z

Cited by 38 publications

(35 citation statements)

References 49 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…For instance, [13] concludes that "3 reviews per paper-i.e., the number generally used in peer review I conferences-give highly confident results … only for conferences with high agreement among reviewers." Our data shows that SIGITE reviewers do not have high agreement, and we therefore recommend having a minimum of four reviewers per paper in the future.…”

Section: Discussionmentioning

confidence: 99%

“…However, other research in this area also found no correlation between reviewer ratings and subsequent academic impact. It is important to remember that, "the aim of the peer review process is not the selection of high impact papers, but is simply to filter junk papers and accept only the ones above a certain quality threshold" [13].…”

Section: Discussionmentioning

confidence: 99%

“…Subsequent examinations of SIGCSE and ITiCSE reviewer data [20] found in contrast that the nationality of reviewers and authors had a small but statistically significant impact on ratings and also acceptance rates. More recently, [13] examined a much larger data set containing over 9000 reviews from ten different conferences in computer science and found quite low reliability and validity measures.…”

Section: Related Workmentioning

confidence: 99%

“…Some have argued that a review process is valid if it selects the best contributions [13]. While this may be the goal of all peer review, it is difficult to measure objectively.…”

Section: Peer Review Validitymentioning

confidence: 99%

See 3 more Smart Citations

A longitudinal examination of SIGITE conference submission data, 2007-2012

Miller

Friedman

2014

Proceedings of the 15th Annual Conference on Information Technology Education

View full text Add to dashboard Cite

This paper examines submission data for the SIGITE conference between the years 2007-2012 with an emphasis on the reliability and validity of the peer review process. Despite the centrality of peer review to academic endeavor, it is not easy to assess it due to the confidentiality of blind reviewing systems. This paper provides a unique addition to the study of peer review by examining reviews and submissions for a single international computing conference across an extended time period. It examines which external factors (such as nationality and familiarity of the reviewer) and which internal characteristics of the submissions (such as word length, number of references, and readability measures) are related to eventual reviewer ratings. Ramifications of the findings for future authors and conference organizers are also discussed.

show abstract

Section: Discussionmentioning

confidence: 99%

Section: Discussionmentioning

confidence: 99%

Section: Related Workmentioning

confidence: 99%

“…Some have argued that a review process is valid if it selects the best contributions [13]. While this may be the goal of all peer review, it is difficult to measure objectively.…”

Section: Peer Review Validitymentioning

confidence: 99%

See 2 more Smart Citations

A longitudinal examination of SIGITE conference submission data, 2007-2012

Miller

Friedman

2014

Proceedings of the 15th Annual Conference on Information Technology Education

View full text Add to dashboard Cite

show abstract

“…The CS community has discussed its conference-centric publishing culture, especially on the subject of review system and article quality (Birman & Schneider, 2009;Fortnow, 2009;Ragone, Mirylenka, Casati, & Marchese, 2013). However, the tradition of holding conference publications in high regard has been established as a de facto norm by the practice of computer scientists for decades, and has been even legitimized as a formal method of evaluating CS scholars for hiring, promotion, and tenure (Franceschet, 2010;Montesi & Owen, 2008;Vardi, 2009).…”

Section: Introductionmentioning

confidence: 99%

Author‐based analysis of conference versus journal publication in computer science

Kim

2018

Asso for Info Science & Tech

View full text Add to dashboard Cite

Conference publications in computer science (CS) have attracted scholarly attention due to their unique status as a main research outlet, unlike other science fields where journals are dominantly used for communicating research findings. One frequent research question has been how different conference and journal publications are, considering an article as a unit of analysis. This study takes an author‐based approach to analyze the publishing patterns of 517,763 scholars who have ever published both in CS conferences and journals for the last 57 years, as recorded in DBLP. The analysis shows that the majority of CS scholars tend to make their scholarly debut, publish more articles, and collaborate with more coauthors in conferences than in journals. Importantly, conference articles seem to serve as a distinct channel of scholarly communication, not a mere preceding step to journal publications: coauthors and title words of authors across conferences and journals tend not to overlap much. This study corroborates findings of previous studies on this topic from a distinctive perspective and suggests that conference authorship in CS calls for more special attention from scholars and administrators outside CS who have focused on journal publications to mine authorship data and evaluate scholarly performance.

show abstract

Commonly Used Indexes for Assessment of Research Production

Vitanov

2016

Qualitative and Quantitative Analysis of Scientific and Scholarly Communication

View full text Add to dashboard Cite

On peer review in computer science: analysis of its effectiveness and suggestions for improvement

Cited by 38 publications

References 49 publications

A longitudinal examination of SIGITE conference submission data, 2007-2012

A longitudinal examination of SIGITE conference submission data, 2007-2012

Author‐based analysis of conference versus journal publication in computer science

Commonly Used Indexes for Assessment of Research Production

Contact Info

Product

Resources

About