An Item Response Theory Evaluation of a Language-Independent CS1 Knowledge Assessment

Abstract

Tests serve an important role in computing education, measuring achievement and differentiating between learners with varying knowledge. But tests may have flaws that confuse learners or may be too difficult or easy, making test scores less valid and reliable. We analyzed the Second Computer Science 1 (SCS1) concept inventory, a widely used assessment of introductory computer science (CS1) knowledge, for such flaws. The prior validation study of the SCS1 used Classical Test Theory and was unable to determine whether differences in scores were a result of question properties or learner knowledge. We extended this validation by modeling question difficulty and learner knowledge separately with Item Response Theory (IRT) and performing expert review on problematic questions. We found that three questions measured knowledge that was unrelated to the rest of the SCS1, and four questions were too difficult for our sample of 489 undergrads from two universities.

Development and Validation of the Middle Grades Computer Science Concept Inventory (MG-CSCI) Assessment

“…FCS1 was developed as a concept inventory for an introductory undergraduate programming course with the goal of assessing all of the core concepts covered in the course (Tew & Guzdial, 2017). The SCS1 assessment (Parker, Guzdial & Engelman, 2016) was a revision and revalidation of the FCS1, though even through this refinement process other researchers continue to question the high difficulty level of some of the items (Xie, Davidson, Li, & Ko, 2019). At the K-12 level, there has been work on developing assessments of student CS conceptual understanding.…”

“…For item development around block-based programming code, prior work on high-school and 4 / 24 middle-school assessments were used to guide our work (Du Boulay, 1986;Shneiderman, & Mayer, 1979, Weintrop & Wilensky, 2015Xie et al, 2019). For each concept, we designed items with varying levels of difficulty and question types: comprehension of a code snippet, debugging a partially wrong code snippet, and developing/completing a partially built code snippet.…”

Fostering Program Comprehension in Novice Programmers - Learning Activities and Learning Trajectories

“…Many studies have investigated students' performance using the FCS1/SCS1 (e.g., [394,256,270,363]). One main advantage of using a validated instrument is to produce comparable and generalizable results, providing meaningful information about the investigated context.…”

“…We found similar student performance on the SCS1 and similar metrics of the SCS1's internal reliability as earlier studies. We now seek to provide a more nuanced analysis to investigate if our data suggest difficulties in the same questions by conducting an Item Response Theory (IRT) analysis using previous research, mainly Xie et al [394] and Parker et al [270], as benchmarks for methods and results.…”

STOIC: Streaming operating systems in the cloud

“…We were guided in our selection of statistical methods by our external evaluator, another evaluator she referred us to, and by recent work in CS education on other CIs. We referred to Chasteen et al [6] as a general guide for how to statistically validate a CI, and then consulted recent work by Xie et al [33] as an example of validating a computer science CI. Guided by both of these works, we applied both Classical Test Theory (CTT) and Item Response Theory (IRT).…”