The process of developing a rubric to assess the cognitive complexity of student-generated multiple choice questions in medical education

Abstract

Cognitively complex assessments encourage students to prepare using deep learning strategies rather than surface learning, recallbased ones. In order to prepare such assessment tasks, it is necessary to have some way of measuring cognitive complexity. In the context of a student-generated MCQ writing task, we developed a rubric for assessing the cognitive complexity of MCQs based on Bloom's taxonomy. We simplified the six-level taxonomy into a three-level rubric. Three rounds of moderation and rubric development were conducted, in which 10, 15 and 100 randomly selected student-generated MCQs were independently rated by three academic staff. After each round of marking, inter-rater reliability was calculated, qualitative analysis of areas of agreement and disagreement was conducted, and the markers discussed the cognitive processes required to answer the MCQs. Inter-rater reliability, defined by the intra-class correlation coefficient, increased from 0.63 to 0.94, indicating the markers rated the MCQs consistently. The three-level rubric was found to be effective for evaluating the cognitive complexity of MCQs generated by medical students.

Medical students create multiple-choice questions for learning in pathology education: a pilot study

“…To assess feasibility of MCQ-writing, we rated MCQs for cognitive complexity and asked students how they went about completing the MCQ task. We rated question quality using a three-level rubric based on Bloom’s taxonomy (summarized in Table 1 , see [ 33 ] for development of the rating system). We also asked students to indicate how long it took them to complete the task, and asked free-text questions (see Table 2 ).…”

The Utility of Multiple-Choice Assessment in Current Medical Education: A Critical Review

“…Inter-rater reliability measures the consistency of student outcomes when tested by different examiners and may reduce in MCQs as a result of varying experience of assessment creators in knowledge and question-writing. Test-retest refers to the reproducibility of consistent MCQ results over time [ 31 ]. MCQs have higher test-retest reliability compared to other methods of assessment such as OSCEs, supervised learning events or essays as the questioning environment is controlled and the options that the candidates can choose are discrete [ 32 ].…”

“…MCQs have higher test-retest reliability compared to other methods of assessment such as OSCEs, supervised learning events or essays as the questioning environment is controlled and the options that the candidates can choose are discrete [ 32 ]. Nevertheless, test-retest reliability can be reduced by having short durations between MCQ tests, as participants may recall information from the first test or too long, as participants could have changed in some way (for example, having life-stressors or change in motivation), which could also bias results [ 31 ]. Internal consistency reliability can be maintained by ensuring consistent difficulty levels between MCQ items comprising the assessment and, again, is generally higher than other forms of assessment [ 9 ].…”

Using Online Wikis for Assessment Transforms Student Articulation of Learning

“…The learning objectives provide a foundation to gauge conceptually planned learning through a process called course mapping (21,22,23 To facilitate student learning and evaluate the complexity of learning, the use of rubrics offers clarity, improved transparency and consistency (25). In addition, the use of rubrics enhances the provision of meaningful feedback to the student for understanding how learning objectives are met (26).…”