Automated Scoring of Constructed‐Response Science Items: Prospects and Obstacles

Liu, Ou Lydia; Brew, Chris; Blackmore, John; Gerard, Libby; Madhok, Jacquie; Linn, Marcia C.

doi:10.1111/emip.12028

Cited by 80 publications

(67 citation statements)

References 33 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…Although science practices such as explanation and argumentation are carried out by means of language, extended text responses written by students are not frequently utilized to inform student learning in science classrooms (Kuo & Wu, ). Automated text scoring can make a real‐time assessment of extended text responses possible without the involvement of extensive time and effort (Liu et al, ; Sherin, ). Automated text scoring occurs when a text is assigned a score on a predetermined scale through various statistical and natural language processing (NLP) techniques.…”

Section: Literature Reviewmentioning

confidence: 99%

“…The scales can be established dichotomously (0–1) or in multiple scoring levels such as 0–2 or 0–5. As originally developed in language assessment, automated text scoring can be used to assess the content of the writing piece (Liu et al, ; Nehm, Ha, & Mayfield, ) or the quality or structure of the overall writing (Bridgeman, Trapani, & Attali, ). The noted benefits of automated text scoring include a reduction in time and effort as well as an establishment of consistency by eliminating human bias.…”

Section: Literature Reviewmentioning

confidence: 99%

See 1 more Smart Citation

Automated text scoring and real‐time adjustable feedback: Supporting revision of scientific arguments involving uncertainty

et al. 2019

Self Cite

View full text Add to dashboard Cite

This paper describes HASbot, an automated text scoring and real‐time feedback system designed to support student revision of scientific arguments. Students submit open‐ended text responses to explain how their data support claims and how the limitations of their data affect the uncertainty of their explanations. HASbot automatically scores these text responses and returns the scores with feedback to students. Data were collected from 343 middle‐ and high‐school students taught by nine teachers across seven states in the United States. A mixed methods design was applied to investigate (a) how students’ utilization of HASbot impacted their development of uncertainty‐infused scientific arguments; (b) how students used feedback to revise their arguments, and (c) how the current design of HASbot supported or hindered students’ revisions. Paired sample t tests indicate that students made significant gains from pretest to posttest in uncertainty‐infused scientific argumentation, ES = 1.52 SD, p < 0.001. Linear regression analysis results indicate that students' HASbot use significantly contributed to their posttest performance on uncertainty‐infused scientific argumentation while gender, English language learner status, and prior computer experience did not. From the analysis of videos, we identified several affordances and limitations of HASbot.

show abstract

Section: Literature Reviewmentioning

confidence: 99%

Section: Literature Reviewmentioning

confidence: 99%

Automated text scoring and real‐time adjustable feedback: Supporting revision of scientific arguments involving uncertainty

et al. 2019

Self Cite

View full text Add to dashboard Cite

show abstract

“…The most recent evaluation of the c-rater approach ) took advantage of some refinements introduced by Sukkarieh and Bolge (2008). O. L. Liu et al (2014) concluded that c-rater cannot replace human scores, although it has shown promise for use in lowstakes settings.…”

Section: Automated Scoringmentioning

confidence: 99%

A Historical Survey of Research Regarding Constructed-Response Formats

Bejar

2017

Methodology of Educational Measurement and Assessment

View full text Add to dashboard Cite

This chapter chronicles ETS research and development contributions related to the use of constructed-response item formats. 1 The use of constructed responses in testing dates back to imperial China, where tests were used in the selection of civil servants. However, in the United States, the multiple-choice format became dominant during the twentieth century, following its invention and use by the SAT ® examinations created by the College Board in 1926. When ETS was created in 1947, post-secondary admissions testing was largely based on tests consisting of multiplechoice items. However, from the start, there were two camps at ETS: those who believed that multiple-choice tests were sufficiently adequate for the purpose of assessing "verbal" skills and those who believed that "direct" forms of assessment requiring written responses had a role to play. For constructed-response formats to regain a foothold in American education several hurdles would need to be overcome. Research at ETS was instrumental in overcoming those hurdles.The first hurdle was that of reliability, specifically the perennial issue of low interrater agreement, which plagued the acceptance of constructed-response formats for most of the twentieth century. The second hurdle was broadening the conception of validity to encompass more than predictive considerations, a process that began with the introduction of construct validity by Cronbach and Meehl (1955). Samuel Messick at ETS played a crucial role in this process by making construct validity relevant to educational tests. An inflexion point in the process of reincorporating constructed-response formats more widely in educational tests was marked 1 Constructed responses to a prompt or question can range in scope and complexity. Perhaps the most common constructed response is the written essay. However, short written responses to questions are also considered to be constructed, as are spoken answers in response to a prompt, mathematical responses (equations, plotted functions, etc.), computer programs, and graphical responses such as architectural designs.

show abstract

“…MCAD systems address part of a complex problem [Carberry and McKenna, 2014] and are complex software tools that require not only a thorough understanding of the various functionalities provided by the system, but also the application of efficient strategies to create high quality CAD models. Based on related research, this issue can be considered a particular type of the "content-based constructed responses" problem [Liu et al, 2014].…”

Section: Supplementary Backgroundmentioning

confidence: 99%

“…Computer Aided Design (CAD) systems are complex software tools whose learning requires correctly managing the functionality provided by the system while applying proper strategies to create high quality CAD models ("content-based constructed responses" [Liu et al, 2014]). CAD models are a relevant part of industrial product design, which is a complex process that requires evaluation [Bailey and Szabo, 2006] and [Waks et al, 2011].…”

Section: Introductionmentioning

confidence: 99%

A Contribution to Conveying Quality Criteria in Mechanical CAD Models and Assemblies through Rubrics and Comprehensive Design Intent Quantification

Otey¹

View full text Add to dashboard Cite

Automated Scoring of Constructed‐Response Science Items: Prospects and Obstacles

Abstract: Content-based automated scoring has been applied in a variety

Cited by 80 publications

References 33 publications

Automated text scoring and real‐time adjustable feedback: Supporting revision of scientific arguments involving uncertainty

Automated text scoring and real‐time adjustable feedback: Supporting revision of scientific arguments involving uncertainty

A Historical Survey of Research Regarding Constructed-Response Formats

A Contribution to Conveying Quality Criteria in Mechanical CAD Models and Assemblies through Rubrics and Comprehensive Design Intent Quantification

Contact Info

Product

Resources

About