The quandary of manuscript reviewing

Abstract

A growing interest in and concern about the adequacy and fairness of modern peer-review practices in publication and funding are apparent across a wide range of scientific disciplines. Although questions about reliability, accountability, reviewer bias, and competence have been raised, there has been very little direct research on these variables.The present investigation was an attempt to study the peer-review process directly, in the natural setting of actual journal referee evaluations of submitted manuscripts. As test materials we selected 12 already published research articles by investigators from prestigious and highly productive American psychology departments, one article from each of 12 highly regarded and widely read American psychology journals with high rejection rates (80%) and nonblind refereeing practices.With fictitious names and institutions substituted for the original ones (e.g., Tri-Valley Center for Human Potential), the altered manuscripts were formally resubmitted to the journals that had originally refereed and published them 18 to 32 months earlier. Of the sample of 38 editors and reviewers, only three (8%) detected the resubmissions. This result allowed nine of the 12 articles to continue through the review process to receive an actual evaluation: eight of the nine were rejected. Sixteen of the 18 referees (89%) recommended against publication and the editors concurred. The grounds for rejection were in many cases described as "serious methodological flaws." A number of possible interpretations of these data are reviewed and evaluated.

Contributions of reviewer judgements to editorial decision-making for the Canadian Journal of Behavioural Science: 1985–1986.

“…The correlation coefficients for interrater agreement are remarkably similar to those reported previously (Gottfredson, 1978; Marsh & Ball, 1981; Whitehurst, 1982). The lowest interrater agreement was found for the importance of a given paper.…”

“…In other studies of interrater agreement, intra-class correlations of different reviewers' appraisals of the same manuscript were computed to determine agreement. These usually have resulted in correlations ranging from .20 to .40 (Gottfredson, 1978; Marsh & Ball, 1981; Whitehurst, 1982). Apart from intra-class coefficient correlations.…”

Interrater agreement for journal manuscript reviews.

“…The literature on peer review in all scholarly fields yields only four reports that include the raw data on agreement (Crandall, 1978;Mahoney, 1977;Peters & Ceci, 1982;Scarr & Weber, 1978), augmented here (Table 2) by previously unreported raw data from Developmental Review and the Merrill-Palmer Quarterly (cf. Whitehurst, 1982). The Peters and Ceci data are not relevant for the present purpose because of a very small N and complete agreement between reviewers, and the Mahoney data represent a special case in that they were not derived from the usual review process.…”

“…Scott (1974) reported an RI of .26 for the Journal of Personality and Social Psychology. Whitehurst (1982) reported an R t of .29 across two related journals, Developmental Review and Merrill-Palmer Quarterly. Peters and Ceci (1982) noted that "Data of this type can certainly erode satisfaction with and confidence in the peer-review system."…”

“…Issues of reviewer prejudice or bias based on author's sex or institutional affiliation aside (e.g., Peters & Ceci, 1982), attempts to improve the peer review process also depend on measures of interrater agreement. Whitehurst (1982Whitehurst ( , 1983, Gottfredson (1978), and Marsh and Ball (1981), for example, used interrater agreement data to evaluate the effectiveness of providing reviewers with extended rating scales based on specific evaluative criteria. Different journals can also be compared using agreement data (Cicchetti, 1980), as can different fields of inquiry (Cole & Cole, 1979).…”