Multiple choice questions can be designed or revised to challenge learners’ critical thinking

Abstract

Multiple choice (MC) questions from a graduate physiology course were evaluated by cognitive-psychology (but not physiology) experts, and analyzed statistically, in order to test the independence of content expertise and cognitive complexity ratings of MC items. Integration of higher order thinking into MC exams is important, but widely known to be challenging-perhaps especially when content experts must think like novices. Expertise in the domain (content) may actually impede the creation of higher-complexity items. Three cognitive psychology experts independently rated cognitive complexity for 252 multiple-choice physiology items using a six-level cognitive complexity matrix that was synthesized from the literature. Rasch modeling estimated item difficulties. The complexity ratings and difficulty estimates were then analyzed together to determine the relative contributions (and independence) of complexity and difficulty to the likelihood of correct answers on each item. Cognitive complexity was found to be statistically independent of difficulty estimates for 88 % of items. Using the complexity matrix, modifications were identified to increase some item complexities by one level, without affecting the item's difficulty. Cognitive complexity can effectively be rated by non-content experts. The six-level complexity matrix, if applied by faculty peer groups trained in cognitive complexity and without domain-specific expertise, could lead to improvements in the complexity targeted with item writing and revision. Targeting higher order thinking with MC questions can be achieved without changing item difficulties or other test characteristics, but this may be less likely if the content expert is left to assess items within their domain of expertise.

A protocol for the development of a critical thinking assessment tool for nurses using a Delphi technique

“…Multiple-choice questions (MCQ) require students to use critical thinking skills to evaluate different response options and determine which best solves the problem (Kerkman & Johnson, 2014). Integration of critical thinking in MCQ is acknowledged to be challenging but can be achieved if principles of higher order thinking are incorporated by question developers (Tractenberg, Gushta, Mulroney, & Weissinger 2013). As suggested by Tractenberg et al (2013), development of the MCQs will be undertaken through collaboration between an assessment expert with specific training in question writing and a clinical nurse with expertise in nursing.…”

“…Integration of critical thinking in MCQ is acknowledged to be challenging but can be achieved if principles of higher order thinking are incorporated by question developers (Tractenberg, Gushta, Mulroney, & Weissinger 2013). As suggested by Tractenberg et al (2013), development of the MCQs will be undertaken through collaboration between an assessment expert with specific training in question writing and a clinical nurse with expertise in nursing. Rasch modelling will be used to determine the reliability and validity of the assessment tool and as a method of estimating item difficulties to assist with scoring of questions (Tractenberg et al, 2013).…”

Cognitive Diagnostic Analysis of Reading Comprehension Items: The Case of English Proficiency Assessment in Saudi Arabia

“…Furthermore, for persons with abilities within the range of practical interest here (-4 ≤ θ ≤ 4), there are more than 50% chances for correct performance of at least two attributes. On the other hand, the cutting θ-scores on the logit scale at which a person has more than 50% chances to perform correctly at least 3, 4, 5, 6, or 7 attributes are (a) θ ≈ -0.5 for at least 3, (b) θ ≈ 0.5 for at least 4, (c) θ ≈ 1.0 for at least 5, (d) θ ≈ 1.5 for at least 6, and (e) θ ≈ 2.0 for at least 7 attributes, respectively [65][66][67][68][69].…”

Development and validation of a written credentialing examination for overseas‐educated dietitians

“…In line with recommendations for professional competency assessments, the majority of items included a ‘stem’ (also known as a ‘scenario’ or ‘vignette’) with relevant clinical or contextual information, which candidates needed to read and understand in order to respond meaningfully to the questions . The use of such a stem enabled the format to assess higher levels of cognition beyond simple recall of isolated facts, such as application of knowledge in different contexts and more integrated clinical reasoning . The items were written so that the key should be identifiable as the best or most defensible option by knowledgeable candidates.…”

“…4 The use of such a stem enabled the format to assess higher levels of cognition beyond simple recall of isolated facts, such as application of knowledge in different contexts and more integrated clinical reasoning. [22][23][24] The items were written so that the key should be identifiable as the best or most defensible option by knowledgeable candidates. The incorrect options, referred to as 'distractors', were designed to present plausible alternatives in the context of the material and question but would be able to be identified as incorrect by candidates with sufficient knowledge and understanding.…”