Comparison of integrated testlet and constructed-response question formats

Abstract

Constructed-response (CR) questions are a mainstay of introductory physics textbooks and exams. However, because of the time, cost, and scoring reliability constraints associated with this format, CR questions are being increasingly replaced by multiple-choice (MC) questions in formal exams. The integrated testlet (IT) is a recently developed question structure designed to provide a proxy of the pedagogical advantages of CR questions while procedurally functioning as set of MC questions. ITs utilize an answer-until-correct response format that provides immediate confirmatory or corrective feedback, and they thus allow not only for the granting of partial credit in cases of initially incorrect reasoning, but, furthermore, the ability to build cumulative question structures. Here, we report on a study that directly compares the functionality of ITs and CR questions in introductory physics exams. To do this, CR questions were converted to concept-equivalent ITs, and both sets of questions were deployed in midterm and final exams. We find that both question types provide adequate discrimination between stronger and weaker students, with CR questions discriminating slightly better than the ITs. There is some indication that any difference in discriminatory power may result from the baseline score for guessing that is inherent in MC testing. Meanwhile, an analysis of interrater scoring of the CR questions raises serious concerns about the reliability of the granting of partial credit when this traditional assessment technique is used in a realistic (but nonoptimized) setting. Furthermore, we show evidence that partial credit is granted in a valid manner in the ITs. Thus, together with consideration of the vastly reduced costs of administering IT-based examinations compared to CR-based examinations, our findings indicate that ITs are viable replacements for CR questions in formal examinations where it is desirable both to assess concept integration and to reward partial knowledge, while efficiently scoring examinations.

Integrated Testlets: A New Form of Expert-Student Collaborative Testing

“…[1,0,0,0,0]). We find that compared to dichotomous scoring, adding any reasonable partial credit scheme slightly boosts the average score, while maintaining the highly-discriminating nature of a good test (Slepkov & Shiell, 2014). Certainly, we have seen no evidence that partial credit dilutes the discriminatory power of the test.…”

“…Moreover, immediate feedback has been demonstrated to improve learning outcomes relative to the results observed with delayed feedback (Dihoff, Brosvic, Epstein, & Cook, 2004). Figure 1 shows a research-validated integrated testlet (Slepkov & Shiell, 2014) specifically designed to test higher-level thinking in a first-year Introductory Physics course. The topic is that of mechanics, and involves the understanding and application of the vector nature of forces, determining friction, Newton's second law, and one-and twodimensional kinematics, with items aligned to particular learning outcomes of the course.…”

“…As part of a formal comparison between IT and free-response formats in exams within an Introductory Physics class we deployed a set of concept-equivalent ITs and free-response questions and found both formats to be both highly discriminating and reliable (Slepkov & Shiell, 2014). A purely psychometrics-based analysis suggested that the free-response format was marginally better at both these measures, but further analysis exposed a large inter-rater variability with the free-response format scoring, while also suggesting that the range of marks awarded for free-response was artificially dispersed, with students between the top and bottom cohorts receiving scores that were only weakly proportional to their mastery of the material.…”

“…There is some preliminary evidence that such a conversation takes place during IT-based examinations. In our previous study (Slepkov & Shiell, 2014) we surveyed students after an Introductory Physics midterm exam that contained two ITs and two free-response questions, each of which covered independent and different topics. We asked students "For the multiple-choice parts of the midterm (i.e.…”

Class Tournament as an Assessment Method in Physics Courses: A Pilot Study

“…However, in different fields a few researchers have come up with an idea of carrying out assessment in some alternative manners (Dochy et al, 1999;Rebello, 2011;Schuwirth & Vleuten, 2004). These include, in particular, different forms of a written test, extensively described and compared in the literature, such as free-and multipleresponse tests (Wilcox & Pollock, 2014), concept tests (such as the Test of Understanding Graphs in Kinematics (Maries & Singh, 2013), Force Concept Inventory (Hestenes et al, 1992) or Brief electricity and magnetism assessment (Ding et al, 2006) and others (Hitt et al, 2014;, constructed-response tests (Slepkov & Shiell, 2014), essay tests (Kruglak, 1955), laboratory skills tests (Doran et al, 1993) and others. Also, many modifications and extensions of these tests have already been proposed in the literature, improving upon their original form (Ding, 2014;Docktor et al, 2015;Wooten et al, 2014;Zwolak & Manogue, 2015).…”