The relationship between essay reading style and scoring proficiency in a psychometric scoring system

Wolfe, Edward W.

doi:10.1016/s1075-2935(97)80006-2

Cited by 70 publications

(84 citation statements)

References 12 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…However, it is difficult to identify all criteria beforehand and in a concrete manner. In addition, various studies have shown that often assessors start with holistic evaluations, which inform their scoring with regard to individual criteria (Cumming et al, 2002;Sakyi, 2003;Vaughan, 1991;Wolfe, 1997). Moreover, when a text does not fit these criteria perfectly, assessors often adjust their scores (Lumley, 2002).…”

Section: Introductionmentioning

confidence: 99%

When teachers compare argumentative texts: Decisions informed by multiple complex aspects of text quality

Lesterhuis¹,

Daal²,

Gasse³

et al. 2018

L1 Educational Studies in Language and Literature

View full text Add to dashboard Cite

Insight into aspects that guide teachers' decisions when assessing student text quality is crucial to an understanding of the validity of text scores. Such research has been lacking in the context of comparative methods which are, however, increasingly being used for text assessment purposes. This study reports on the aspects of argumentative texts that guide Flemish teachers' decisions when using the comparative judgement method. In using this method, teachers indicate which text of a pair is of higher quality. 27 teachers explained 23 comparative judgements in a decision statement, when comparing randomly selected texts written by 135 students in their fifth year of general secondary education. This resulted in 596 statements referring to 2054 segments of aspects of text quality. Firstly, an inductive analysis revealed that teachers consider a wide range of aspects with regard to text quality when making comparison decisions. Secondly, the deductive aggregation of these aspects showed that most decisions are informed by the organisation and argumentation of the texts. Lastly, almost all statements reported complex aspects of text quality, whereas half of the decision statements also showed a reflection on the rule-applying aspects of text quality. We conclude that comparative judgement encourages teachers to make decisions on complex and multiple aspects of text quality. Further research should elaborate on whether the aspects that informed teachers' decisions are related to the text they choose, and whether teachers differ in the aspects to which they refer.

show abstract

Section: Introductionmentioning

confidence: 99%

When teachers compare argumentative texts: Decisions informed by multiple complex aspects of text quality

Lesterhuis¹,

Daal²,

Gasse³

et al. 2018

L1 Educational Studies in Language and Literature

View full text Add to dashboard Cite

show abstract

“…Descriptions of the qualifications of raters have also been proposed (Powers et al 1998;). In addition, the nature of scoring expertise has been studied (Wolfe 1997;Wolfe et al 1998). Methods to capture and monitor rater effects during scoring as a function of rater characteristics are similarly relevant Myford and Mislevy 1995;Patz et al 2002).…”

Section: The Emergence Of a Solutionmentioning

confidence: 99%

A Historical Survey of Research Regarding Constructed-Response Formats

Bejar

2017

Methodology of Educational Measurement and Assessment

View full text Add to dashboard Cite

This chapter chronicles ETS research and development contributions related to the use of constructed-response item formats. 1 The use of constructed responses in testing dates back to imperial China, where tests were used in the selection of civil servants. However, in the United States, the multiple-choice format became dominant during the twentieth century, following its invention and use by the SAT ® examinations created by the College Board in 1926. When ETS was created in 1947, post-secondary admissions testing was largely based on tests consisting of multiplechoice items. However, from the start, there were two camps at ETS: those who believed that multiple-choice tests were sufficiently adequate for the purpose of assessing "verbal" skills and those who believed that "direct" forms of assessment requiring written responses had a role to play. For constructed-response formats to regain a foothold in American education several hurdles would need to be overcome. Research at ETS was instrumental in overcoming those hurdles.The first hurdle was that of reliability, specifically the perennial issue of low interrater agreement, which plagued the acceptance of constructed-response formats for most of the twentieth century. The second hurdle was broadening the conception of validity to encompass more than predictive considerations, a process that began with the introduction of construct validity by Cronbach and Meehl (1955). Samuel Messick at ETS played a crucial role in this process by making construct validity relevant to educational tests. An inflexion point in the process of reincorporating constructed-response formats more widely in educational tests was marked 1 Constructed responses to a prompt or question can range in scope and complexity. Perhaps the most common constructed response is the written essay. However, short written responses to questions are also considered to be constructed, as are spoken answers in response to a prompt, mathematical responses (equations, plotted functions, etc.), computer programs, and graphical responses such as architectural designs.

show abstract

“…• Provide training on how to score compositions (Shohamy, Gordon, & Kraemer, 1992) ' Use experienced raters to score papers (Schoonen, Verger, & Eiting, 1997;Underwood & Murphy, 1998;Wolfe, 1997;Wolfe, Kao, Ranney, 1998;Wolfe & Ranney, 1996) ' Have multiple raters score each paper (Bürgin & Hughes, 2009;Coffman, 1966;Gearhart, Herman, Novak, & Wolf, 1995;Godshalk, Swineford, & Coffman, 1966;Swartzetal., 1999). ' Base students' writing score on multiple writing tasks (Bürgin & Hughes, 2009;Finlayson, 1951;Godshalk Swineford, & Coffman, 1966;Lehman, 1990). '…”

Section: Make High-stakes Writing Assessments Fairmentioning

confidence: 99%

Throw 'em Out or Make 'em Better? State and District High-Stakes Writing Assessments

Graham¹,

Hebert²,

Harris³

2017

Exceptional

View full text Add to dashboard Cite

The relationship between essay reading style and scoring proficiency in a psychometric scoring system

Cited by 70 publications

References 12 publications

When teachers compare argumentative texts: Decisions informed by multiple complex aspects of text quality

When teachers compare argumentative texts: Decisions informed by multiple complex aspects of text quality

A Historical Survey of Research Regarding Constructed-Response Formats

Throw 'em Out or Make 'em Better? State and District High-Stakes Writing Assessments

Contact Info

Product

Resources

About