Relationships between Bloom’s taxonomy, judges’ estimation of item difficulty and psychometric properties of items from a progress test: a prospective observational study

Abstract

BACKGROUND: Progress tests are longitudinal assessments of students' knowledge based on successive tests. Calibration of the test difficulty is challenging, especially because of the tendency of item-writers to overestimate students' performance. The relationships between the levels of Bloom's taxonomy, the ability of test judges to predict the difficulty of test items and the real psychometric properties of test items have been insufficiently studied. OBJECTIVE: To investigate the psychometric properties of items according to their classification in Bloom's taxonomy and judges' estimates, through an adaptation of the Angoff method. DESIGN AND SETTING: Prospective observational study using secondary data from students' performance in a progress test applied to ten medical schools, mainly in the state of São Paulo, Brazil. METHODS: We compared the expected and real difficulty of items used in a progress test. The items were classified according to Bloom's taxonomy. Psychometric properties were assessed based on their taxonomy and fields of knowledge. RESULTS: There was a 54% match between the panel of experts' expectations and the real difficulty of items. Items that were expected to be easy had mean difficulty that was significantly lower than that of items that were expected to be medium (P < 0.05) or difficult (P < 0.01). Items with high-level taxonomy had higher discrimination indices than low-level items (P = 0.026). We did not find any significant differences between the fields in terms of difficulty and discrimination. CONCLUSIONS: Our study demonstrated that items with high-level taxonomy performed better in discrimination indices and that a panel of experts may develop coherent reasoning regarding the difficulty of items.

Assessment of factual recall and higher-order cognitive domains in an open-book medical school examination

“…Due to considerable overlap between boundaries, low numbers of ‘Apply’ questions and no questions from levels 4–6 of Bloom’s taxonomy, we combined ‘Understand’ and ‘Apply’ questions to produce a binary predictor of ‘Remember’ versus ‘Understand/Apply’. We recognize that some evidence suggests no linear relationship between taxonomic level and item difficulty (Cunnington et al, 1996 ; Hamamoto Filho et al, 2020 ) and there is limited evidence to support Bloom’s taxonomy as a single, linear hierarchy in this way (Furst, 1981 ; Haladyna & Rodriguez, 2013 ; Marzano & Kendall, 2007 ). However, our main hypothesis was that factual recall questions would be made selectively easier in an OBE, so we considered collapsing the categories appropriate to compare, in effect, recall questions versus questions that required more than recall.…”

Item’s Analysis in Health Education at the High School of Health Science of Tunis

“…Recently, it was reported that question items with high-level taxonomy performed better in discrimination indices. [30] Ideal questions reached only 32% yet higher levels were reported in literature 47% [31].…”

Exploring pooled analysis of pretested items to monitor the performance of medical students exposed to different curriculum designs

“…Items are multiple choice questions with four options and a single correct answer. Preferably, the items are clinical vignette-based aiming for applied knowledge rather than knowledge recall [25]. In 2019, four existing consortia in São Paulo state developed the exam by selecting the best pre-tested items (tested between 2013 and 2018 and preferably, between 2016 and 2018, with good discrimination indices) conforming to the commonly used blueprint.…”

“…In this regard, the philosophy of the new curriculum (i.e., more integration between the basic sciences and social sciences applicable to medicine) may support the performance of students in basic sciences and public health. Notably, students exposed to the new curriculum integrating basic and clinical sciences may be better prepared for the interinstitutional Progress Test [31], which uses high taxonomy vignette-based items [25]. In addition, experiences in community settings may also contribute to early medical education [32].…”