Resolving Score Differences in the Rating of Writing Samples: Does Discussion Improve the Accuracy of Scores?

Johnson, Robert L.; Penny, James; Gordon, Belita; Shumate, Steven R.; Fisher, Steven P.

doi:10.1207/s15434311laq0202_2

Cited by 50 publications

(32 citation statements)

References 27 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…We compared the nature of the novice discussion to that of the experts with particular regard to the themes that emerged. To increase reliability, two different researchers coded the transcripts and met to discuss their codes until 100% agreement was attained (following that described in Johnson, Penny, Gordon, Shumate, & Fisher, 2005). These researchers followed a similar protocol to interpret the results.…”

Section: Methodsmentioning

confidence: 99%

Laboratory materials: Affordances or constraints?

et al. 2011

View full text Add to dashboard Cite

Laboratory instruction is critical to the understanding of biology and is a central piece of biological sciences instruction. Although much investigation has focused on the content of biology laboratory exercises, we contend that understanding the extent to which the laboratory materials can aid or limit experimental investigation is of equal importance. In this study, therefore, we investigate the role of timing and availability of laboratory equipment in the context of two different laboratory exercises. We use both case study and an experimental approach to investigate how laboratory materials guide the planning, context, creativity, and timing of ideas shared among students. Our data support the notion that providing students with laboratory equipment before students plan and consider different experimental approaches can constrain students' ideas and encourage tool-focused solutions to experimental design tasks. ß

show abstract

Section: Methodsmentioning

confidence: 99%

Laboratory materials: Affordances or constraints?

et al. 2011

View full text Add to dashboard Cite

show abstract

“…Apply a two-step scoring process where the rater matches the composition to the closest benchmark and then scores it again if it does not match this benchmark perfectly by adding a plus or minus to the first score (Penny, Johnson, Gordon, 2000a, 2000bJohnson, Penny, Fisher, & Kuhs, 2003). ' Have raters discuss and resolve differences in their scores (Johnson, Penny, Gordon, Shumate, & Fisher, 2005). ' Combine the scores of two disagreeing raters with a third score provided by a more experienced rater (Johnson, Penny, & Gordon, 2000.…”

Section: Make High-stakes Writing Assessments Fairmentioning

confidence: 99%

Throw 'em Out or Make 'em Better? State and District High-Stakes Writing Assessments

Graham¹,

Hebert²,

Harris³

2017

Exceptional

View full text Add to dashboard Cite

“…Una de las críticas más frecuentes a los estudios que intentan medir las habilidades de escritura tiene relación con la confiabilidad de las evaluaciones, pues el lenguaje es siempre objeto de interpretación. Aparte de la calidad de la escritura, múltiples fuentes de error pueden contribuir a la variabilidad de los puntajes, entre ellas, diferencias de criterio entre correctores, ambigüedad de los criterios de corrección y variaciones de las condiciones en que ésta se realiza [15][16][17] .…”

Section: N V E S T I G a C I ó Nunclassified