Validation of automated scoring of science assessments

Liu, Ou Lydia; Rios, Joseph A.; Heilman, Michael; Gerard, Libby; Linn, Marcia C.

doi:10.1002/tea.21299

Cited by 99 publications

(81 citation statements)

References 33 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…Sao Pedro, Baker, Gobert, Montalvo, and Nakama () described a set of “detectors” that could fairly reliably detect whether a student was performing controlled or uncontrolled experiments, whether the student's experiment was testing a hypothesis or not, and whether students were planning their behaviour (see also Gobert, Kim, Sao Pedro, Kennedy, & Betts, ; Gobert, Sao Pedro, Raziuddin, & Baker, ). Liu, Rios, Heilman, Gerard, and Linn () assessed the validity of a tool called c‐rater‐ML for the automatic scoring of students' open‐ended responses and found good agreement between the automatic and human expert ratings.…”

Section: The Next Steps In Technology‐based Guidance Of the Inquiry Pmentioning

confidence: 99%

Moving towards engaged learning in STEM domains; there is no simple answer, but clearly a road ahead

Jong

2019

Computer Assisted Learning

View full text Add to dashboard Cite

What is the best approach to educating students is, evidently, the pivotal question in educational research. In the general debate on this question, clear positions are often taken―for example, whether teacher‐led instruction or more student‐directed approaches should be followed. In the current narrative review, this central question and the different stances taken are explored for a specific field of learning, STEM (science, technology, engineering, and mathematics) domains. The current article starts with an argument as to why the traditional, more directive teacher‐led educational approach to STEM learning may not always lead to deep conceptual knowledge and proposes exploring more engaged forms of learning. More specifically, a particular arrangement for engaged learning, inquiry learning, is taken as an example in this exploration. It is argued that the effects of educational methods are influenced by a multitude of (interacting) factors related to student and teacher characteristics, context, domain, frequency of use, timing, and probably more. Therefore, this article tries to outline a more nuanced and balanced stance towards choosing educational approaches. A picture is presented to show that adopting and combining different approaches, at both the lesson and the curricular levels, may be necessary to reach optimal levels of learning. Existing and upcoming technologies may play a decisive role in realizing this more complex, but also more realistic view of learning in STEM.

show abstract

Section: The Next Steps In Technology‐based Guidance Of the Inquiry Pmentioning

confidence: 99%

Moving towards engaged learning in STEM domains; there is no simple answer, but clearly a road ahead

Jong

2019

Computer Assisted Learning

View full text Add to dashboard Cite

show abstract

“…Liu, Rios, Heilman, Gerard, and Linn () identified 11 studies where automated text scoring assessed students’ responses to open‐ended science items. Of the 11 studies, nine addressed college students’ writing and two studies addressed middle‐school students’ writing.…”

Section: Literature Reviewmentioning

confidence: 99%

Automated text scoring and real‐time adjustable feedback: Supporting revision of scientific arguments involving uncertainty

et al. 2019

Self Cite

View full text Add to dashboard Cite

This paper describes HASbot, an automated text scoring and real‐time feedback system designed to support student revision of scientific arguments. Students submit open‐ended text responses to explain how their data support claims and how the limitations of their data affect the uncertainty of their explanations. HASbot automatically scores these text responses and returns the scores with feedback to students. Data were collected from 343 middle‐ and high‐school students taught by nine teachers across seven states in the United States. A mixed methods design was applied to investigate (a) how students’ utilization of HASbot impacted their development of uncertainty‐infused scientific arguments; (b) how students used feedback to revise their arguments, and (c) how the current design of HASbot supported or hindered students’ revisions. Paired sample t tests indicate that students made significant gains from pretest to posttest in uncertainty‐infused scientific argumentation, ES = 1.52 SD, p < 0.001. Linear regression analysis results indicate that students' HASbot use significantly contributed to their posttest performance on uncertainty‐infused scientific argumentation while gender, English language learner status, and prior computer experience did not. From the analysis of videos, we identified several affordances and limitations of HASbot.

show abstract

“…In our early trials we could reach average correlations of 0.37 between the prediction of abilities in a criterion and later-earned scores in an exam in this topic [24]. In contrast, the best correlation between a human rater and an automatic scoring system is about r=0.52 [25]. Given the fact that the LMSA Kit uses educational data stored in the LMS to predict results in a future exam, the smaller correlations are not surprising.…”

Section: The Lmsa Kit Offers Insights Into Students' Learningmentioning

confidence: 99%

Translating Numbers into Feedback: Providing Students with Automatically Generated Feedback

Hedtrich¹,

Graulich²

2018

EDUCATION

View full text Add to dashboard Cite

Universities nowadays are confronted with the challenge of offering students sufficient formative feedback about their learning progress. Undergraduates in particular struggle in this almost impersonal learning situation, in contrast to their experience at school. Web-based trainings often do not allow a comparable formative feedback from a teacher, as current Learning Management Systems only offer basal feedback mechanisms. The Learning Management System Analyzation Kit and the Easy Snippet Feedback Edit are software tools that have been developed to overcome this shortcoming. The intuitive design of the software allows teachers to develop small feedback generating programs, which use the educational data of EDM solutions, raw data of the LMS, or just teachers' individual records to compose individual formative feedback messages for all students. Initial evaluations show that students appreciate this new type of feedback, and that even inexperienced users can easily operate the software.

show abstract

Validation of automated scoring of science assessments

Cited by 99 publications

References 33 publications

Moving towards engaged learning in STEM domains; there is no simple answer, but clearly a road ahead

Moving towards engaged learning in STEM domains; there is no simple answer, but clearly a road ahead

Automated text scoring and real‐time adjustable feedback: Supporting revision of scientific arguments involving uncertainty

Translating Numbers into Feedback: Providing Students with Automatically Generated Feedback

Contact Info

Product

Resources

About