Probing the question order effect while developing a chemistry concept inventory

Abstract

The design of assessment tools is critical to accurately evaluate students' understanding of chemistry. Although extensive research has been conducted on various aspects of assessment tool design, few studies in chemistry have focused on the impact of the order in which questions are presented to students on the measurement of students' understanding and students' performance. This potential impact has been labeled the question order effect in other literature and may be considered as a threat to the construct validity of the assessment tool. The set of studies described in this article tested whether question order effects were present within a concept inventory on acid-based chemistry. In particular, we tested whether the order of two conceptually isomorphic questions, one pictorial and one verbal, affected students' performance on the concept inventory. Two different versions of the inventory were developed and collected from students enrolled in the second semester of first-year university chemistry courses (general chemistry;N= 774) at two different institutions and to students enrolled in the first semester of organic chemistry (N= 163) at one of the two institutions. Students were further divided in two groups based on their self-reported level of effort in answering the concept inventory. Interviews were also conducted with a total of 19 students at various stages of the studies. Analyses of differences in students' responses to the two versions of the inventory revealed no question order effect in all settings. Implications for instructors and researchers are provided.

Does question order matter on online math assessments? A big data analysis of undergraduate mathematics final exams

“…and Undersander et al (2017) found that it mattered whether verbal or pictorial question were presented first.…”

Concept Inventories as a Complement to Learning Progressions

“…Although originally used mainly at the introductory level, CIs have increasingly been developed for more advanced-level topics in the molecular life sciences (Smith et al, 2008;Shi et al, 2010;Bretz and Linenberger, 2012;Couch et al, 2015;McFarland et al, 2017). The shortcomings of CIs have been documented when used simply as multiple-choice pre-and posttests (Coppola and Krajcik, 2014), and the nature and order of the questions has been shown to activate different levels of learning (McClary and Talanquer, 2011;Warfa and Odowa, 2015;Undersander et al, 2017). However, they are more likely to probe deep understanding than other types of assessment applied to large groups (Xu et al, 2017) and can be adapted to be more open-ended and, hence, more informative (Dood et al, 2018).…”

“…, 2017 ). The shortcomings of CIs have been documented when used simply as multiple-choice pre- and posttests ( Coppola and Krajcik, 2014 ), and the nature and order of the questions has been shown to activate different levels of learning ( McClary and Talanquer, 2011 ; Warfa and Odowa, 2015 ; Undersander et al , 2017 ). However, they are more likely to probe deep understanding than other types of assessment applied to large groups ( Xu et al.…”

Passport of Geology Museum: a Description Analysis on Pupils’ Worksheet

“…This passport is in the form of a book containing the pupil's identity along with a column for expressing opinions. These columns are also filled with short probing questions (Undersander, Lund, Langdon, and Stains, 2017). Pupils can also not only write their opinions through writing but also through pictures.…”