Data-Driven Investigation into Variants of Code Writing Questions

Abstract

To defend against collaborative cheating in code writing questions, instructors of courses with online, asynchronous exams can use the strategy of question variants. These question variants are manually written questions to be selected at random during exam time to assess the same learning goal. In order to create these variants, currently the instructors have to rely on intuition to accomplish the competing goals of ensuring that variants are different enough to defend against collaborative cheating, and yet similar enough where students are assessed fairly. In this paper, we propose data-driven investigation into these variants. We apply our data-driven investigation into a dataset of three midterm exams from a large introductory programming course. Our results show that (1) observable inequalities of student performance exist between variants and (2) these differences are not just limited to score. Our results also show that the information gathered from our data-driven investigation can be used to provide recommendations for improving design of future variants.

Reexamining Learning Curve Analysis in Programming Education: The Value of Many Small Problems

“…Isomorphic questions are question variants designed to test the same programming concepts while modifying surfacelevel details [4,18]. Isomorphic questions have been used in computing education both to assess learning and expand the size of question pools for assessments [5,38,26,21,11].…”

Issues of Question Equivalence in Online Exam Pools

“…Research on the effective design of question pools is concentrated in the field of standardized testing (American Educational Research Association, American Psychological Association, & National Council on Measurement in Education, 2014). Additional work has been done in computer science courses (Butler et al, 2020;Denny et al, 2019;Sud et al, 2019) and in examining statistical factors that affect question duplication among students (Murdock & Brenneman, 2020). In this study, we specifically analyzed pool types across science courses for variation in student performance.…”