How Item Writers Understand Depth of Knowledge

Abstract

An important part of test development is ensuring alignment between test forms and content standards. One common way of measuring alignment is the Webb (1997Webb ( , 2007 alignment procedure. This article investigates (a) how well item writers understand components of the definition of Depth of Knowledge (DOK) from the Webb alignment procedure and (b) how consistent their DOK ratings are with ratings provided by other committees of educators across grade levels, content areas, and alternate assessment levels in a Midwestern state alternate assessment system. Results indicate that many item writers understand key features of DOK. However, some item writers struggled to articulate what DOK means and had some misconceptions. Additional analyses suggested some lack of consistency between the item writer DOK ratings and the committee DOK ratings. Some notable differences were found across alternate assessment levels and content areas. Implications for future item writing training and alignment studies are provided.

Using Assessments to Investigate and Compare the Nature of Learning in Undergraduate Science Courses

“…The weak or missing relationship between cognitive skill level and student performance contradicts a common misconception in education at the K–12 and undergraduate level, namely, that cognitive complexity directly correlates with item difficulty (Lord and Baviskar, 2007; Wyse and Viger, 2011). Evidence from studies directly probing the relationship between cognitive complexity and difficulty underscores the limited or nonexistent relationship between these two constructs (Wyse and Wyse, personal communication; Nehm and Schonfeld, 2008).…”

The micro revolution: effect of Bite-Sized Teaching (BST) on learner engagement and learning in postgraduate medical education

“…Limitations of our work reliance on resident self-reported impact on attitudes and behaviors and multiple choice test to assess learning. While we used a validated test, this does not adequately capture impact on higher level learning processes [ 34 ]. It is possible that score differences relate to differences between groups or the impact of repeated testing effects rather than learning through BST, although the patterns in scores and demographic variables between groups does not suggest this.…”

Questions for Assessing Higher-Order Cognitive Skills: It's Not Just Bloom’s

“…In informal conversations with biologists prior to this study, we have noticed that difficulty is often discussed. Additionally, another study, focused on K–12 item writers, showed that item writers conflated cognitive level and difficulty (Wyse and Viger, 2011). We suspect that our prediction would bear out in further studies.…”