Estimation of ability from homework items when there are missing and/or multiple attempts

Bergner, Yoav; Colvin, Kimberly F.; Pritchard, David E.

doi:10.1145/2723576.2723582

Cited by 13 publications

(14 citation statements)

References 17 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…A built in assumption to such an approach is that higher ability students are likely to arrive at a correct response in fewer attempts than lower ability students. Earlier studies have confirmed this assumption empirically and shown that polytomous models can indeed result in more informative estimates than dichotomous models based only on first attempts (Bergner, Colvin, & Pritchard, ). Alternatively, multiple‐attempt information may come into play by modeling “recovery” after an incorrect attempt as a separate skill dimension (Culpepper, ).…”

mentioning

confidence: 78%

See 1 more Smart Citation

Item Response Models for Multiple Attempts With Incomplete Data

Bergner

Choi

Castellano

2019

J Educational Measurement

Self Cite

View full text Add to dashboard Cite

Allowance for multiple chances to answer constructed response questions is a prevalent feature in computer-based homework and exams. We consider the use of item response theory in the estimation of item characteristics and student ability when multiple attempts are allowed but no explicit penalty is deducted for extra tries. This is common practice in online formative assessments, where the number of attempts is often unlimited. In these environments, some students may not always answeruntil-correct, but may rather terminate a response process after one or more incorrect tries. We contrast the cases of graded and sequential item response models, both unidimensional models which do not explicitly account for factors other than ability. These approaches differ not only in terms of log-odds assumptions but, importantly, in terms of handling incomplete data. We explore the consequences of model misspecification through a simulation study and with four online homework data sets. Our results suggest that model selection is insensitive for complete data, but quite sensitive to whether missing responses are regarded as informative (of inability) or not (e.g., missing at random). Under realistic conditions, a sequential model with similar parametric degrees of freedom to a graded model can account for more response patterns and outperforms the latter in terms of model fit.

show abstract

mentioning

confidence: 78%

“…As described in Bergner et al. (), total scores often have low variance and pronounced ceiling effects when multiple attempts are allowed. We thus use proportion correct on first attempt, c 1 , as a classical measure of ability.…”

Section: Empirical Studymentioning

confidence: 94%

Item Response Models for Multiple Attempts With Incomplete Data

Bergner

Choi

Castellano

2019

J Educational Measurement

Self Cite

View full text Add to dashboard Cite

show abstract

“…On the other hand, learners' performance is approached by visual feedback and analysis of graduate attributes that engages students with self‐assessment as part of their personal and professional development (Thompson, ). Furthermore, Okada and Tada () define a method for formative assessment specialized in experience‐based activities performed in real‐world learning sceneries; Xing, Wadholm, and Goggins () deploy an automated strategy to assess participation in a multimode discourse environment, where a clustering algorithm is applied according to the Activity Theory principles; Vogelsang and Ruppertz () apply a grading system, Cloud Teaching Assistant System, for automated validation in MOOCs, as well as an additional tool to instructor and peer grading; Vahdati, Lange, and Auern () assess the quality of open courses in terms of fitness to identify subjects to improve; Bergner, Colvin, and Pritchard () deal with a couple of issues related with scoring of student item response data from MOOCs, large amounts of missing data, and allowances for multiple attempts to answer.…”

Section: Applicationsmentioning

confidence: 99%

Learning analytics: A glance of evolution, status, and trends according to a proposed taxonomy

Peña-Ayala

2018

WIREs Data Min & Knowl

View full text Add to dashboard Cite

Before the emergence of computer‐based educational systems (CBES) whose aims of providing teaching and learning experiences to hundreds even thousands of users, an explosion of information (e.g., students' log data) demands sophisticated methods to gather, analyze, and interpret learners' traces to regulate and enhance education. Thus, learning analytics (LA) arises as a knowledge discovery paradigm that provides valuable findings and facilitates stakeholders to understand the learning process and its implications. Therefore, a landscape of the LA nature, its underlying factors, and applications achieved is outlined in this paper according to a suggested LA Taxonomy that classifies the LA duty from a functional perspective. The aim is to provide an idea of the LA toil, its research lines, and trends to inspire the development of novel approaches for improving teaching and learning practices. Furthermore, the scope of this review covers recently published papers in prestigious journals and conferences, where the works dated from 2016 are summarized and those corresponding to 2014–2015 are cited according to the proposed LA taxonomy. A glimpse is sketched of LA, where underlying elements frame the field foundations to ground the approaches. Moreover, LA strengths, weaknesses, challenges, and risks are highlighted to advice how the LA arena could be enhanced and empowered. In addition, this review offers an insight of the recent LA labor, as well as motivates readers to enrich the LA achievements. This work promotes the LA practice giving an account of the job being achieved and reported in literature, as well as a reflection of the state‐of‐the‐art and an acumens vision to inspire future labor. This article is categorized under: Application Areas > Education and Learning Application Areas > Science and Technology Fundamental Concepts of Data and Knowledge > Human Centricity and User Interaction

show abstract

“…Moreover, they can be applied even with a large number of missing values. The potential of using item response theory in LA has been shown, for example, by Bergner et al (2015), who estimated student abilities based on homework scores from an MOOC in which a large number of scores were missing.…”

Section: Characteristics and Forms Of The Pisa Datamentioning

confidence: 99%

Knowledge Discovery from the Programme for International Student Assessment

Saarela

Kärkkäinen

2017

Learning Analytics: Fundaments, Applications, and Trends

View full text Add to dashboard Cite

The Programme for International Student Assessment (PISA) is a worldwide study that assesses the proficiencies of 15-year-old students in reading, mathematics, and science every three years. Despite the high quality and open availability of the PISA data sets, which call for big data learning analytics, academic research using this rich and carefully collected data is surprisingly sparse. Our research contributes to reducing this deficit by discovering novel knowledge from the PISA through the development and use of appropriate methods. Since Finland has been the country of most international interest in the PISA assessment, a relevant review of the Finnish educational system is provided. This chapter also gives a background on learning analytics and presents findings from a novel case study. Similar to the existing literature on learning analytics, the empirical part is based on a student model; however, unlike in the previous literature, our model represents a profile of a national student population. We compare Finland to other countries by hierarchically clustering these student profiles from all the countries that participated in the latest assessment and validating the results through statistical testing. Finally, an evaluation and interpretation of the variables that explain the differences between the students in Finland and those of the remaining PISA countries is presented. Based on our analysis, we conclude that, in global terms, learning time and good student-teacher relations are not as important as collaborative skills and humility to explain students' success in the PISA test.

show abstract

Estimation of ability from homework items when there are missing and/or multiple attempts

Cited by 13 publications

References 17 publications

Item Response Models for Multiple Attempts With Incomplete Data

Item Response Models for Multiple Attempts With Incomplete Data

Learning analytics: A glance of evolution, status, and trends according to a proposed taxonomy

Knowledge Discovery from the Programme for International Student Assessment

Contact Info

Product

Resources

About