On the Validity of Machine Learning-based Next Generation Science Assessments: A Validity Inferential Network

Abstract

This study provides a solid validity inferential network to guide the development, interpretation, and use of machine learningbased next-generation science assessments (NGSAs). Given that machine learning (ML) has been broadly implemented in the automatic scoring of constructed responses, essays, simulations, educational games, and interdisciplinary assessments to advance the evidence collection and inference of student science learning, we contend that additional validity issues arise for science assessments due to the involvement of ML. These emerging validity issues may not be addressed by prior validity frameworks developed for either non-science or non-ML assessments. We thus examine the changes brought in by ML to science assessments and identify seven critical validity issues of ML-based NGSAs: potential risk of misrepresenting the construct of interest, potential confounders due to that more variables may involve, nonalignment between interpretation and use of scores and designed learning goals, nonalignment between interpretation and use of scores and actual learning quality, nonalignment between machine scores and rubrics, limited generalizable ability of machine algorithmic models, and limited extrapolating ability of machine algorithmic models. Based on the seven validity issues identified, we propose a validity inferential network to address the cognitive, instructional, and inferential validity of ML-based NGSAs. To demonstrate the utility of this network, we present an exemplar of ML-based next-generation science assessments that was developed using a seven-step ML framework. We articulate how we used the validity inferential network to ensure accountable assessment design, as well as valid interpretation and use of machine scores.

Practices and Theories: How Can Machine Learning Assist in Innovative Assessment Practices in Science Education

“…To meet the vision, science educators have to engage students in practices to improve students' competence to construct explanations, figure out solutions, and solve problems. The articles in this special issue made substantial contributions by tapping into science learning that is embedded with such complex scientific practices such as modeling (Zhai et al, 2020c), scientific argumentation (Lee et al, 2021;Wang et al, 2020), investigation (Maestrales et al, 2021), multimodal representational thinking (Sung et al, 2020), explanation (Jescovitch et al, 2020), and epistemic knowledge of model-based explanation (Rosenberg & Krist, 2020). For example, in their study, Maestrales et al (2021) employed ML to automatically score students' performance by the dimension of science learning and achieved high scoring accuracy.…”

“…This is because multiple-choice items are difficult to elicit higherorder thinking that is associated with sophisticated cognitions and performance. The articles in this special issue extended the approaches to collecting evidence in ways such as virtual reality (e.g., Sung et al, 2020), representations (e.g., Zhai et al, 2020c), and facial expression identification (e.g., Liaw et al, 2020). In their study, Sung et al (2020) employed the augmented reality technology with a thermal camera attached to a smartphone to elicit students' understanding and asked students to write constructed responses; students' responses were analyzed using a deep learning approach.…”

“…In their study, Sung et al (2020) employed the augmented reality technology with a thermal camera attached to a smartphone to elicit students' understanding and asked students to write constructed responses; students' responses were analyzed using a deep learning approach. Zhai et al (2020c) demonstrated how to apply deep learning to automatically evaluate students' modeling competency by scoring their drawing and writing explanations. Liaw et al (2020) also employed a deep learning-based app to automatically identify students' emotion and examined how their emotion was associated with their learning outcomes.…”

“…Interestingly, both studies yielded consistent conclusions: machines generated more accurate results by using the analytic scoring approach as compared with using the holistic approach. Zhai et al (2020c) specifically examined the validity issues of ML-based next generation science assessments and proposed a validity inferential network to guide assessment development and validity conclusions made using ML scores.…”

“…After reviewing and editing this special issue, I believe that we are getting closer thanks to the contributions of the articles. As compared with a recent review study (Zhai et al, 2020e), the articles in this special issue employed more matured deep learning algorithms in assessing drawing (Zhai et al, 2020c), epistemic knowledge (Rosenberg & Krist, 2020), and higher-order thinking in science (Lamb et al, 2020;Sung et al, 2020). I realized that the approaches to collecting evidence were also improved because of ML.…”