Indicadores de gestão na revisão por pares: confiabilidade da revisão recíproca anônima de propostas de mestrado │ Peer review indicators: reliability of a reciprocal anonymous review of masters proposals

Abstract

Resumo A revisão por pares é pobre em indicadores, apesar de sua importância. Este artigo apresenta indicadores de confiabilidade da revisão recíproca anônima de propostas de mestrado. Doze alunos de mestrado em Ciência da Informação deram pareceres anônimos sobre propostas de colegas em 7 itens avaliativos, usando escala Likert-6, sendo a confiabilidade calculada através de índices utilizados na literatura. A confiabilidade não é uma medida absoluta de qualidade, mas o cálculo desses indicadores permite estudar sistematicamente a qualidade da revisão por pares.Palavras-chave Revisão por pares, Confiabilidade, Concordância entre revisores, Coeficiente de correlação intraclasse, Revisão por pares na aprendizagem. Peer review indicators: reliability of a reciprocal anonymous review of masters proposalsAbstract Peer review lacks indicators, which is incongruent with its importance. This article reports the reliability of mutual, anonymous review of masters research proposals. Twelve masters students in Information Science refereed anonymously their peers' research proposals according to 7 evaluation items, using a Likert-6 scale. Reliability was expressed as intraclass correlation indices between 0.500 and 0.202, with 9 positive and 3 negative values -lower than usual in professional processes. Reliability is not an absolute measure of quality, but calculating reliability allows for a systematic study of the quality of peer review.

A proposal of formative assessment in EFL teaching and learning: Online writing and peer-review activities

“…In an analogous manner, the ICC (consistency) varies from -2.246 to .928, according to Table 12. These are substantial results when compared with Denisczwicz and Kern (2013)'s results (-.062 to .261). The ICC (consistency) reveals the level of consistency among reviewers.…”

“…The Median for the ICC (agreement) is .348 and for the ICC (consistency) is .471 It is much higher than Denisczwicz and Kern (2013)'s Medians for ICC (agreement) of .058 and ICC (consistency) of .097. They are even higher than Weller's Median ICC of .30 (no specific ICC provided) from a study that analyzed reviews from professional researchers (Weller 2002, in Denisczwicz & Kern, 2013.…”

“…Reliability, according to Silverman (p. 224, in: Dornyei, 2007, p. 57), "refers to the degree of consistency with which instances are assigned to the same category by different observers or by the same observer on different occasions". The more adequate tests to measure Reliability between reviewers, according to Denisczwicz and Kern (2013) are the Intraclass Correlation Coefficient ICC (agreement) and the ICC (consistency), being the former more suitable than the latter. As they reveal Reliability from different perspectives, statistics for both types were run employing the statistics program SPSS v.24.…”