A Historical Survey of Research Regarding Constructed-Response Formats

Abstract

This chapter chronicles ETS research and development contributions related to the use of constructed-response item formats. 1 The use of constructed responses in testing dates back to imperial China, where tests were used in the selection of civil servants. However, in the United States, the multiple-choice format became dominant during the twentieth century, following its invention and use by the SAT ® examinations created by the College Board in 1926. When ETS was created in 1947, post-secondary admissions testing was largely based on tests consisting of multiplechoice items. However, from the start, there were two camps at ETS: those who believed that multiple-choice tests were sufficiently adequate for the purpose of assessing "verbal" skills and those who believed that "direct" forms of assessment requiring written responses had a role to play. For constructed-response formats to regain a foothold in American education several hurdles would need to be overcome. Research at ETS was instrumental in overcoming those hurdles.The first hurdle was that of reliability, specifically the perennial issue of low interrater agreement, which plagued the acceptance of constructed-response formats for most of the twentieth century. The second hurdle was broadening the conception of validity to encompass more than predictive considerations, a process that began with the introduction of construct validity by Cronbach and Meehl (1955). Samuel Messick at ETS played a crucial role in this process by making construct validity relevant to educational tests. An inflexion point in the process of reincorporating constructed-response formats more widely in educational tests was marked 1 Constructed responses to a prompt or question can range in scope and complexity. Perhaps the most common constructed response is the written essay. However, short written responses to questions are also considered to be constructed, as are spoken answers in response to a prompt, mathematical responses (equations, plotted functions, etc.), computer programs, and graphical responses such as architectural designs.

The Teaching Mode Design and Effect Evaluation Method of Animation Course From the Perspective of Big Data

“…For teachers, improving data literacy is both one of the components of improving personal literacy and one of the main aspects of promoting data literacy education practices (Bejar, 2017). According to Professor Gurman of American Psychology, data literacy is the ability of teachers to understand and use data effectively and to make decisions and implement them accordingly.…”

Evaluación de la escritura a gran escala: análisis de la prueba de escritura del Tercer Estudio Regional Comparativo y Explicativo (TERCE)

“…Dos aproximaciones han predominado en la evaluación de la escritura a gran escala: la directa y la indirecta (Huot, 2002). La primera supone que la evaluación debe basarse en el desempeño del estudiante al construir un texto escrito, por lo que se realiza con pruebas que solicitan producir textos completos e implica la participación de codificadores; la indirecta se realiza con pruebas de selección de respuesta, lo cual, para algunos investigadores, tiene la ventaja de excluir la subjetividad de los codificadores, y reducir el tiempo y costo que demanda el análisis de textos escritos (Bejar, 2017).…”

“…Diversos estudios advierten dificultades en el análisis de los textos escritos, como diferencias de interpretación de las rúbricas entre los codificadores según su formación previa y experiencia, y el modo y plazo en los que fueron formados; sesgos de indulgencia o tendencia a dejar pasar errores y de severidad o tendencia a ver más errores de los existentes; falta de consistencia del codificador a lo largo del proceso de codificación o privilegio de la variedad lingüística dominante en desmedro de otras variedades (Bejar, 2017;Behizadeh, 2014;Mendoza, 2018). Para asegurar la calidad de la codificación, usualmente se procura controlar el grado de acuerdo entre los codificadores.…”

Scoring Graphical Responses in TIMSS 2019 Using Artificial Neural Networks

“…The transition to a computer-based assessment of student skills has enabled international large-scale assessments (ILSAs) to improve measurement through innovative item-types including constructed-response items. Not only are constructed-response items considered more engaging than traditional item types (e.g., multiple-choice items), but they also capture complex constructs with greater nuance and precision (Bejar, 2017; Zhai, 2021). The Trends in International Mathematics and Science Study (TIMSS) 2019 digital assessment, for example, incorporated many constructed-response items, including grid-based image responses (graphing and drawing), to better measure higher-order mathematics and science skills (Cotter et al, 2020).…”