Is it harder to know or to reason? Analyzing two-tier science assessment items using the Rasch measurement model

Abstract

Two-tier multiple-choice (TTMC) items are used to assess students' knowledge of a scientific concept for tier 1 and their reasoning about this concept for tier 2. But are the knowledge and reasoning involved in these tiers really distinguishable? Are the tiers equally challenging for students? The answers to these questions influence how we use and interpret TTMC instruments. We apply the Rasch measurement model on TTMC items to see if the items are distinguishable according to different traits (represented by the tier), or according to different content sub-topics within the instrument, or to both content and tier. Two TTMC data sets are analyzed: data from Singapore and Korea on the Light Propagation Diagnostic Instrument (LPDI), data from the United States on the Classroom Test of Scientific Reasoning (CTSR). Findings for LPDI show that tier-2 reasoning items are more difficult than tier-1 knowledge items, across content sub-topics. Findings for CTSR do not show a consistent pattern by tier or by content sub-topic. We conclude that TTMC items cannot be assumed to have a consistent pattern of difficulty by tier-and that assessment developers and users need to consider how the tiers operate when administering TTMC items and interpreting results. Researchers must check the tiers' difficulties empirically during validation and use. Though findings from data in Asian contexts were more consistent, further study is needed to rule out differences between the LPDI and CTSR instruments.Keywords: Science education, Two-tier items, Rasch measurement models, Optics, Scientific reasoning Assessing student learning-of scientific concepts, practices, or habits of mind-is one of the central topics for research and development in science education. Such assessments can serve as formative or diagnostic tools for planning instruction and working with students, or as summative tools for gauging the effectiveness of our instructional practices, curriculum materials, or teacher education efforts. However, we observe an ongoing tension in science education assessment between our ability to construct conventional test items (e.g., multiple choice questions) that can be highly reliable but are perceived to be incapable of providing richer insights into students' conceptions and ways of thinking. Research addressing this includes efforts to make better sense of how students' responses to test items can be understood from a broader view of conceptual understanding, ability, or skill Neumann et al. 2011).One solution proposed that can address this tension are two-tier items (Treagust 1988). The two tiers in two-tier items act together to uncover students' understanding of core concepts because the student must choose a seemingly "factual" knowledge response for the first tier (Taber and Tan 2011), and then choose for the second tier what reasoning about the concept they used to arrive at the first-tier response. A large body of research across contexts has applied two-tier items to uncover students' understanding of scientific ...

Validity evaluation of the Lawson classroom test of scientific reasoning

“…In addition, the multidimensionality of LCTSR has been partially established using confirmatory factor analysis [40]. As a two-tier instrument, the LCTSR is assumed to measure students' knowing in the first tier and their reasoning processes in the second tier [42], and this has been confirmed in a recent empirical study [41].…”

“…Understanding the relation between knowing and reasoning can provide important cognitive and education insights into how teaching and learning can be improved. Recent studies have started to show a possible progression from knowing to reasoning, which suggests that reasoning is harder than knowing [34,41,[53][54][55][56]. However, the actual difficulty reflected within a given test can also be affected by a number of factors including test design, content, and population [34,41], and therefore needs to be evaluated with all the possible factors considered and or controlled.…”

“…Only recently have a few studies started to examine its construct validity [38][39][40][41]. For example, using Rasch analysis and principle component analysis, the assumption of a unidimensional construct for a general scientific reasoning ability has been shown to be acceptable [38,39].…”

“…Since its formal introduction three decades ago [42], TTMC designs have been widely researched and applied in education assessment [44][45][46][47][48][49][50][51][52]. The LCTSR is a well-known example of a two-tier test designed to assess scientific reasoning [11,16,41,42,44].…”

Asia-Pacific Science Education (APSE): expanding opportunities for publishing science education research

“…The researchers found significant differences in the range of scores and discuss the need for additional research design tools that are both domain specific and holistic enough to assess other facets of giftedness. Fulmer et al (2015) share findings from a study using two-tier multiple-choice (TTMC) items to assess students' knowledge of a scientific concept and their reasoning about this concept. Employing a Rasch measurement model on TTMC items, the researchers explored whether items were distinguishable according to different traits, different content sub-topics within the instrument, or both content and tier.…”