Applying machine learning in science assessment: a systematic review

Zhai, Xiaoming; Yin, Yue; Pellegrino, James W.; Haudek, Kevin C.; Shi, Lehong

doi:10.1080/03057267.2020.1735757

Cited by 116 publications

(120 citation statements)

References 58 publications

Supporting

Mentioning

106

Contrasting

Unclassified

Order By: Relevance

“…This work helps address the challenges of using ML in science assessment. We have attempted to advance the application of ML to evaluate CR of a complex construct (Zhai 2019): principle-based reasoning in the content-rich domain of undergraduate physiology. The responses collected in this project are rich in science content, symbols, and relationships among components (e.g., differential movement of two ions), which can be challenges to developing accurate ML models using limited training sets, as is often the case in science assessments.…”

Section: Discussionmentioning

confidence: 99%

“…Additionally, we have attempted to apply ML to a complex construct of a LP (Zhai 2019). LPs represent a model of cognitive development and may have various and complex underlying construct structures (Wilson 2009).…”

Section: Discussionmentioning

confidence: 99%

“…However, CR answers can be difficult to evaluate, interpret, and return feedback in a timely manner for both instructors and students (Gerard et al 2019). Over the last few decades, many studies have reported on the utility of ML in education and assessments as a way of quickly and accurately evaluating written text (Zhai et al 2020).…”

Section: Machine Learning Of Constructed Response Assessmentsmentioning

confidence: 99%

“…A recent review has shown that there is a wide variety of ML algorithms employed in automated scoring of text responses (Zhai et al 2020). Further, the study points out that technical features of ML are often not reported.…”

Section: Challenges In Applying Coding Approaches To Machine Learningmentioning

confidence: 99%

“…Machine learning (ML) has potential for influencing education and has been increasingly applied specifically to science assessments (Kotsiantis 2012). Specifically, one promising aspect of ML is the potential for assessing more complex constructs that are difficult to capture with more traditional assessments (Zhai et al 2020). However, little research has examined how certain approaches for, or technical features of, ML influence the success of assessing more complex constructs.…”

mentioning

confidence: 99%

See 4 more Smart Citations

Comparison of Machine Learning Performance Using Analytic and Holistic Coding Approaches Across Constructed Response Assessments Aligned to a Science Learning Progression

et al. 2020

Self Cite

View full text Add to dashboard Cite

We systematically compared two coding approaches to generate training datasets for machine learning (ML): (i) a holistic approach based on learning progression levels and (ii) a dichotomous, analytic approach of multiple concepts in student reasoning, deconstructed from holistic rubrics. We evaluated four constructed response assessment items for undergraduate physiology, each targeting five levels of a developing flux learning progression in an ion context. Human-coded datasets were used to train two ML models: (i) an 8-classification algorithm ensemble implemented in the Constructed Response Classifier (CRC), and (ii) a single classification algorithm implemented in LightSide Researcher’s Workbench. Human coding agreement on approximately 700 student responses per item was high for both approaches with Cohen’s kappas ranging from 0.75 to 0.87 on holistic scoring and from 0.78 to 0.89 on analytic composite scoring. ML model performance varied across items and rubric type. For two items, training sets from both coding approaches produced similarly accurate ML models, with differences in Cohen’s kappa between machine and human scores of 0.002 and 0.041. For the other items, ML models trained with analytic coded responses and used for a composite score, achieved better performance as compared to using holistic scores for training, with increases in Cohen’s kappa of 0.043 and 0.117. These items used a more complex scenario involving movement of two ions. It may be that analytic coding is beneficial to unpacking this additional complexity.

show abstract

Section: Discussionmentioning

confidence: 99%

Section: Discussionmentioning

confidence: 99%

Section: Machine Learning Of Constructed Response Assessmentsmentioning

confidence: 99%

Section: Challenges In Applying Coding Approaches To Machine Learningmentioning

confidence: 99%

mentioning

confidence: 99%

See 3 more Smart Citations

Comparison of Machine Learning Performance Using Analytic and Holistic Coding Approaches Across Constructed Response Assessments Aligned to a Science Learning Progression

et al. 2020

Self Cite

View full text Add to dashboard Cite

show abstract

Beyond analytics: Using computer‐aided methods in educational research to extend qualitative data analysis

Vieira,

Ortega‐Alvarez,

Magana

et al. 2024

Comp Applic In Engineering

View full text Add to dashboard Cite

This study proposes and demonstrates how computer‐aided methods can be used to extend qualitative data analysis by quantifying qualitative data, and then through exploration, categorization, grouping, and validation. Computer‐aided approaches to inquiry have gained important ground in educational research, mostly through data analytics and large data set processing. We argue that qualitative data analysis methods can also be supported and extended by computer‐aided methods. In particular, we posit that computing capacities rationally applied can expand the innate human ability to recognize patterns and group qualitative information based on similarities. We propose a principled approach to using machine learning in qualitative education research based on the three interrelated elements of the assessment triangle: cognition, observation, and interpretation. Through the lens of the assessment triangle, the study presents three examples of qualitative studies in engineering education that have used computer‐aided methods for visualization and grouping. The first study focuses on characterizing students' written explanations of programming code, using tile plots and hierarchical clustering with binary distances to identify the different approaches that students used to self‐explain. The second study looks into students' modeling and simulation process and elicits the types of knowledge that they used in each step through a think‐aloud protocol. For this purpose, we used a bubble plot and a k‐means clustering algorithm. The third and final study explores engineering faculty's conceptions of teaching, using data from semi‐structured interviews. We grouped these conceptions based on coding similarities, using Jaccard's similarity coefficient, and visualized them using a treemap. We conclude this manuscript by discussing some implications for engineering education qualitative research.

show abstract

Can AI be racist? Color‐evasiveness in the application of machine learning to science assessments

Cheuk

2021

Science Education

View full text Add to dashboard Cite

Assessment developers are increasingly using the developing technology of machine learning in transforming how to assess students in their science learning. I argue that these algorithmic models further embed the structures of inequality that are pervasive in the development of science assessments in how they legitimize certain language practices that protect the hierarchical standing of status quo interests. My argument is situated within the broader emerging ethical challenges around this new technology.I apply a raciolinguistic equity analysis framework in critiquing the "new black box" that reinforces structural forms of discrimination against the linguistic repertoires of racially marginalized student populations. The article ends with me sharing a set of tactical shifts that can be deployed to form a more equitable and socially-just field of machine learning enhanced science assessments.

show abstract

Applying machine learning in science assessment: a systematic review

Cited by 116 publications

References 58 publications

Comparison of Machine Learning Performance Using Analytic and Holistic Coding Approaches Across Constructed Response Assessments Aligned to a Science Learning Progression

Comparison of Machine Learning Performance Using Analytic and Holistic Coding Approaches Across Constructed Response Assessments Aligned to a Science Learning Progression

Beyond analytics: Using computer‐aided methods in educational research to extend qualitative data analysis

Can AI be racist? Color‐evasiveness in the application of machine learning to science assessments

Contact Info

Product

Resources

About