PrairieLearn: Mastery-based Online Problem Solving with Adaptive Scoring and Recommendations Driven by Machine Learning

West, Matthew; Herman, Geoffrey L.; Zilles, Craig

doi:10.18260/p.24575

Cited by 86 publications

(44 citation statements)

References 11 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…and for handling students with testing accommodations to be sufficient. Instructors have quite positive opinions of the custom learning management systems (PrairieLearn [15], [16] and RELATE [7]) that we use in the CBTF, their reliability, and their support for a broad range of question types. Instructors are generally satisfied with the anti-cheating measures that the CBTF provides in both question randomization to prevent collaborative cheating between students taking exams at different times [4] and the physical security provided by the proctors.…”

Section: Faculty Feedbackmentioning

confidence: 99%

Student and Instructor Experiences With a Computer-Based Testing Facility

Zilles¹,

West²,

Mussulman³

et al. 2018

EDULEARN Proceedings

Self Cite

View full text Add to dashboard Cite

In this work, we explore how a large-scale introduction of computer-based testing has impacted students and instructors. The College of Engineering at the University of Illinois has been operating a Computer-Based Testing Facility (CBTF) for four years now, and the CBTF has matured into a fixture of our College. In Fall 2017, the CBTF served 21 courses from seven different departments and over 6,000 unique students. Over 52,000 exams were proctored, including 3,500 final exams. This paper summarizes key findings from a collection of surveys completed by students and instructors. Most instructors report having positive experiences with the CBTF. A large majority report that, once the exam content is in place, they perceive reductions in the effort to run and grade exams and to handle exceptional situations. Instructors also like how the CBTF enables them to run small frequent tests, run second-chance exams, and test computational skills. The vast majority of the surveyed instructors plan to continue using the CBTF and think that it should be expanded. FIXME: Some highlights from our student surveys include: (1) students generally are more satisfied with CBTF exams relative to traditional paper exams (45% satisfied or very satisfied, vs. 17% dissatisfied or very dissatisfied), but this preference seems to vary by major, with computer science and electrical engineering majors even more strongly preferring the computerized exams, (2) students' favorite aspects of CBTF exams include the flexibility to schedule them at convenient times, that CBTF courses generally have more frequent, shorter tests, and the opportunities to take second chance exams, and (3) some students prefer the partial credit mechanism commonly used in traditional written exams, where credit is granted for work shown for incorrect answers.

show abstract

Section: Faculty Feedbackmentioning

confidence: 99%

Student and Instructor Experiences With a Computer-Based Testing Facility

Zilles¹,

West²,

Mussulman³

et al. 2018

EDULEARN Proceedings

Self Cite

View full text Add to dashboard Cite

show abstract

“…Computer-based exams have been proposed as a means of mitigating the tension between scale and excellence in assessment in engineering classes (DeMara et al, 2016;Shacham, 1998;Zilles et al, 2015). Such exams allow a broad range of questions (e.g., numeric, graphical, symbolic, programming, and drawing) to be autograded and to provide students with immediate feedback (Carrasquel, 1985;Rytkönen & Myyry, 2014;Shacham, 1998;West, Herman, & Zilles, 2015). Several studies have demonstrated the validity of computer-based testing across a broad range of subjects (Bodmann & Robinson, 2004;Boevé, Meijer, Albers, Beetsma, & Bosker, 2015;Bugbee Jr., 1996;Cagiltay & Ozalp-Yaman, 2013;McDonald, 2002;Prisacari & Danielson, 2017;Zandvliet & Farragher, 1997).…”

Section: Introductionmentioning

confidence: 99%

Analyzing the decline of student scores over time in self‐scheduled asynchronous exams

Chen

West

Zilles

2019

J of Engineering Edu

Self Cite

View full text Add to dashboard Cite

Background When students are given a choice of when to take an exam in engineering and computing courses, it has been previously observed that average exam scores generally decline over the exam period. This trend may have implications both for the design of interventions to improve student learning and for data analysis to detect collaborative cheating. Purpose/Hypothesis We hypothesize that average exam scores decline over the exam period primarily due to self‐selection effects, where weaker students tend to choose exam times later in the exam period, while stronger students are more likely to choose earlier times. Design/Method We collected 31,673 exam records over four semesters from six undergraduate engineering and computing courses that had both synchronous exams (all students at the same time) and asynchronous exams (students choose a time). We analyzed student exam time choice and asynchronous exam scores, using synchronous exam scores in the same course as a control variable. Results We find that students with lower scores on synchronous exams generally elect to take asynchronous exams later and that controlling for student ability (via synchronous exams) removes 70% of the decline observed in average asynchronous exam scores over the exam period but does not eliminate the downward trend with time. Conclusions We conclude that self‐selection effects are primarily responsible for exam score declines over time, that exam time selection is unlikely to be a useful target for interventions to improve performance, and that there is no evidence for widespread collaborative cheating in the dataset used in this research.

show abstract

“…The system described in Section 3 was implemented within the PrairieLearn 14 online system and used for both homeworks and testing-centere 3;15 quizzes (frequent exams) in an introductory mechanics course at the University of Illinois at Urbana-Champaign in Fall 2016 with 180 students. The PrairieLearn system was introduced in the same class in Fall 2015, however, the drawing tool was not yet available and questions assessing shear and bending moment diagrams were created using a multiple-choice format.…”

Section: Results From Implementationmentioning

confidence: 99%