What errors do peer reviewers detect, and does training improve their ability to detect them?

Abstract

Objective To analyse data from a trial and report the frequencies with which major and minor errors are detected at a general medical journal, the types of errors missed and the impact of training on error detection.Design 607 peer reviewers at the BMJ were randomized to two intervention groups receiving different types of training (face-to-face training or a self-taught package) and a control group. Each reviewer was sent the same three test papers over the study period, each of which had nine major and five minor methodological errors inserted.Setting BMJ peer reviewers. Main outcome measuresThe quality of review, assessed using a validated instrument, and the number and type of errors detected before and after training. ResultsThe number of major errors detected varied over the three papers.The interventions had small effects. At baseline (Paper 1) reviewers found an average of 2.58 of the nine major errors, with no notable difference between the groups.The mean number of errors reported was similar for the second and third papers, 2.71 and 3.0, respectively. Biased randomization was the error detected most frequently in all three papers, with over 60% of reviewers rejecting the papers identifying this error. Reviewers who did not reject the papers found fewer errors and the proportion finding biased randomization was less than 40% for each paper.Conclusions Editors should not assume that reviewers will detect most major errors, particularly those concerned with the context of study. Short training packages have only a slight impact on improving error detection.

Are we witnessing the decline effect in the Type D personality literature? What can be learned?

“…Across fields, there have been a number of suggestions for changes in editorial practices that are intended to make the occurrence of decline effects less common [61][62][63][64][65]. These include some fundamental changes like publication of negative findings and non-replications; prior registration of the protocols for observational studies, much the same as clinical trials are now being registered; making publicly available data sets for reanalysis; and not accepting "discoveries" until they are independently replicated.…”

Revisiones por pares de protocolos presentados a un Comité Local de Investigación en Salud. Tabasco, México, 2009

“…Respecto a los resultados obtenidos, cabe resaltar el hecho de que esta es la primera investigación conocida por el autor al momento del estudio en la que se exploran los errores presentes en los protocolos de investigación, dado que la literatura consultada durante este periodo informa estos errores en otro tipo de productos científicos, tales como resúmenes y reportes de investigación [11][12][13][14][15][16][17][18][19][20][21][22][23][24][25] , así como traducciones de éstos, cuya estructura, extensión y contenidos son diferentes; de modo que los resultados observados en esta serie no son comparables con los referidos en la citada literatura, desconociendo si actualmente existen investigaciones más recientes en las que se haya evaluado el mismo tipo de documentos científicos valorados en este trabajo.…”

“…Como resultado, se ha informado que los errores más frecuentes son manipulación de datos, errores en los criterios de selección, análisis incorrecto de observaciones múltiples y fabricación de datos, entre otros expuestos en la Tabla 1 [11][12][13][14][15][16][17][18][19][20][21][22][23][24][25] . Esta situación de errores metodológicos y estructurales no solo es presentada por las investigaciones de corte cuantitativo en ciencias de la salud, sino que también ha sido referida en investigaciones cualitativas, de uso principal en ciencias sociales 26,27 .…”

Consistency Guarantees for Professional Appraisal of Geotechnical Engineering Design under Market Leading and Deciding

“…3 How to be a noblesse oblige executive leader of peer review group Use of reviewers dilutes the effect of a biased reviewer (Schroter, Black, Evans, Godlee, Osorio, & Smith, 2008). Actually the role of leader of a peer review group is taken by HC.…”