Multilevel Mixture Modeling with Propensity Score Weights for Quasi-Experimental Evaluation of Virtual Learning Environments

Leite, Walter L.; Jing, Zeyuan; Kuang, Huan; Kim, Dongho; Huggins-Manley, Anne Corinne

doi:10.1080/10705511.2021.1919895

Cited by 5 publications

(4 citation statements)

References 73 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Instead of manually selecting the variables, the program evaluator may use the ant colony optimization algorithm instead (Jing et al, 2019). The ant colony optimization algorithm and other methods to automatically select variables may also be helpful when a program evaluator is considering many variables (Leite et al, 2021).…”

Section: Discussionmentioning

confidence: 99%

“…This approach has potential to simultaneously account for uncertainty of latent clusters and nonrandom selection into latent clusters. For more information on links between propensity score analysis and FMM consider (Collier & Leite, 2020;Leite et al, 2021). The DPP method could be incorporated into this analysis because propensity score analysis does not eliminate the model fit challenge with FMM.…”

Section: Discussionmentioning

confidence: 99%

See 1 more Smart Citation

Finite Mixture Modeling for Program Evaluation: Resampling and Pre-processing Approaches

2021

View full text Add to dashboard Cite

Background Finite mixture models cluster individuals into latent subgroups based on observed traits. However, inaccurate enumeration of clusters can have lasting implications on policy decisions and allocations of resources. Applied and methodological researchers accept no obvious best model fit statistic, and different measures could suggest different numbers of latent clusters. Objectives The purpose of this article is to evaluate and compare different cluster enumeration techniques. Research Design Study I demonstrates how recently proposed resampling methods result in no precise number of clusters on which all fit statistics agree. We recommend the pre-processing method in Study II as an alternative. Both studies used nationally representative data on working memory, cognitive flexibility, and inhibitory control. Conclusions The data plus priors method shows promise to address inconsistencies among fit measures and help applied researchers using finite mixture models in the future.

show abstract

Section: Discussionmentioning

confidence: 99%

Section: Discussionmentioning

confidence: 99%

Finite Mixture Modeling for Program Evaluation: Resampling and Pre-processing Approaches

2021

View full text Add to dashboard Cite

show abstract

“…We focused on analyzing responses to items from the entire item bank of each Algebra I domain, which are used in the VLE to randomly draw questions for 10-item quizzes. Student use of these quizzes to practice has been positively related to student achievement on Florida’s high-stakes Algebra 1 End-of-Course assessment (Leite et al, 2019; Leite et al, 2021). For each domain, the number of items in the item bank ranged from 41 to 89, and we assumed that each domain consisted of a unique trait.…”

Section: Exploratory Studymentioning

confidence: 99%

Semisupervised Learning Method to Adjust Biased Item Difficulty Estimates Caused by Nonignorable Missingness in a Virtual Learning Environment

Xue

Huggins-Manley

Leite

2021

Educational and Psychological Measurement

Self Cite

View full text Add to dashboard Cite

In data collected from virtual learning environments (VLEs), item response theory (IRT) models can be used to guide the ongoing measurement of student ability. However, such applications of IRT rely on unbiased item parameter estimates associated with test items in the VLE. Without formal piloting of the items, one can expect a large amount of nonignorable missing data in the VLE log file data, and this is expected to negatively affect IRT item parameter estimation accuracy, which then negatively affects any future ability estimates utilized in the VLE. In the psychometric literature, methods for handling missing data have been studied mostly around conditions in which the data and the amount of missing data are not as large as those that come from VLEs. In this article, we introduce a semisupervised learning method to deal with a large proportion of missingness contained in VLE data from which one needs to obtain unbiased item parameter estimates. First, we explored the factors relating to the missing data. Then we implemented a semisupervised learning method under the two-parameter logistic IRT model to estimate the latent abilities of students. Last, we applied two adjustment methods designed to reduce bias in item parameter estimates. The proposed framework showed its potential for obtaining unbiased item parameter estimates that can then be fixed in the VLE in order to obtain ongoing ability estimates for operational purposes.

show abstract

“…The increasing popularity of online learning has offered massive opportunities and flexibilities to students for accessing quality learning experiences since the last decade. Serving as an important alternative learning format to face-to-face learning, learning online has become ubiquitous during the COVID-19 pandemic (Leite et al., 2021). This learning format makes it possible for students that are geographically dispersed to learn together and provides instructors with detailed digital learning traces that are used to examine students’ learning efforts.…”

Section: Introductionmentioning

confidence: 99%

An Interpretable Pipeline for Identifying At-Risk Students

Pei

Xing

2021

Journal of Educational Computing Research

View full text Add to dashboard Cite

This paper introduces a novel approach to identify at-risk students with a focus on output interpretability through analyzing learning activities at a finer granularity on a weekly basis. Specifically, this approach converts the predicted output from the former weeks into meaningful probabilities to infer the predictions in the current week for maintaining the consecutiveness among learning activities. To demonstrate the efficacy of our model in identifying at-risk students, we compare the weekly AUCs and averaged performance (i.e., accuracy, precision, recall, and f1-score) over each course with the baseline models (i.e., Random Forest, Support Vector Machine, and Decision Tree), respectively. Furthermore, we adopt a Top- K metric to examine the number of at-risk students that the model is able to identify with high precision during each week. Finally, the model output is interpreted through a model-agnostic interpretation approach to support instructors to make informed recommendations for students’ learning. The experimental results demonstrate the capability and interpretability of our model in identifying at-risk students in online learning settings. In addition to that our work also provides significant implications in building accountable machine learning pipelines that can be used to automatically generated individualized learning interventions while considering fairness between different learning groups.

show abstract

Multilevel Mixture Modeling with Propensity Score Weights for Quasi-Experimental Evaluation of Virtual Learning Environments

Cited by 5 publications

References 73 publications

Finite Mixture Modeling for Program Evaluation: Resampling and Pre-processing Approaches

Finite Mixture Modeling for Program Evaluation: Resampling and Pre-processing Approaches

Semisupervised Learning Method to Adjust Biased Item Difficulty Estimates Caused by Nonignorable Missingness in a Virtual Learning Environment

An Interpretable Pipeline for Identifying At-Risk Students

Contact Info

Product

Resources

About