Examinee Cohort Size and Item Analysis Guidelines for Health Professions Education Programs: A Monte Carlo Simulation Study

Aubin, André-Sébastien; Young, Meredith; Eva, Kevin W.; St‐Onge, Christina

doi:10.1097/acm.0000000000002888

Cited by 9 publications

(12 citation statements)

References 14 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…The discrimination index of a MCQ is lower in students with high scores because they choose distractors less frequently than the students with low scores [7]. This nding is also in agreement with a recent paper that shows that the reliability of easy exams is lower than the average of di cult exams reliability [17].…”

Section: Discussionsupporting

confidence: 85%

“…According to our ndings, the lower discriminative power of MCQ exams in groups of students with high scores, could be attenuated increasing the number of case-based questions. Another potential way for correcting this low discriminative power could be to create speci c exams for each group, with different di culty levels, but it is di cult, even for large-scale professional testing organizations, to create exams at a predetermined level of di culty [17]. Furthermore, exams with different di culty levels for students in the same course may vulnerate justice.…”

Section: Discussionmentioning

confidence: 99%

See 1 more Smart Citation

Evaluation of the quality of multiple-choice questions according to the students' academic level

Iñarrairaegui

Fernández-Ros

Lucena

et al. 2022

Preprint

View full text Add to dashboard Cite

Background. One of the most important challenges in medical education is the preparation of multiple-choice questions able to discriminate between students with different academic level. Average questions may be very easy for students with good performance, reducing their discriminant power in this group of students. The aim of this study was to analyze if the discriminative power of multiple-choice questions is different according to the students' academic performance. Methods. We retrospectively analyzed the difficulty and discrimination indices of 257 multiple-choice questions used for the end of course examination of pathophysiology and analyzed whether the discrimination indices were lower in students with good academic performance (group 1) than in students with moderate/poor academic performance (group 2). We also evaluated whether case-based questions maintained their discriminant power better than factual questions in both groups of students or not. Comparison of the difficulty and discrimination indices between both groups was based on the Wilcoxon test.Results. Difficulty index was significantly higher in group 1 (median: 0.78 versus 0.56; P < 0.001) and discrimination index was significantly higher in group 2 (median: 0.21 versus 0.28; P < 0.001). Factual questions had higher discriminative indices in group 2 than in group 1 (median: 0.28 versus 0.20; P < 0.001), but discriminative indices of case-based questions did not differ significantly between groups (median: 0.30 versus 0.24; P=0.296). Conclusions. Multiple-choice question exams have lower discriminative power in the group of students with high scores. The use of clinical vignettes may allow to maintain the discriminative power of multiple-choice questions.

show abstract

Section: Discussionsupporting

confidence: 85%

Section: Discussionmentioning

confidence: 99%

Evaluation of the quality of multiple-choice questions according to the students' academic level

Iñarrairaegui

Fernández-Ros

Lucena

et al. 2022

Preprint

View full text Add to dashboard Cite

show abstract

“…Medical students (n � 211) at the final week of their 5 th semester (having been taught basic pharmacology for the first time in that semester by means of lectures) were given a formative online test in February 2018 (Figure 1). e number of students fits well to recent work on size requirements in assessment studies [16]. We gave students the opportunity to get bonus points from the online test for their summative test a week later.…”

Section: Methodsmentioning

confidence: 66%

Comparison of Online Tests of Very Short Answer versus Single Best Answers for Medical Students in a Pharmacology Course over One Year

Neumann

Simmrodt

Teichert

et al. 2021

Education Research International

View full text Add to dashboard Cite

Single best answers (single choice, SC) are the classical tools used in medical examinations on all levels of education. In contrast, very short answer (VSA) probably requires sound prior knowledge and deeper learning than SC, and VSA should make cueing and guessing impossible. Hence, in a basic pharmacology course, we wanted to compare the SC and VSA formats directly at the end of the course and one year later. Medical students (n = 211) were given a formative online test. Two groups were randomly formed (A and B). Participants in group A were first given fifteen single choices (one out of five) pharmacology questions and thereafter fifteen very short answer questions (open question which were to be answered online and semiautomatically assessed). Very similar questions with regard to learning objectives but in opposite order were given to group B. After one year, about half of students from group A were again given the very same questions (AA) or the opposite questions (AB). Likewise, group B was again tested with the opposite questions (BA) or the same (BB). The SC questions in groups A, AA, AB, B, BA, and BB were in sum easier to answer than the corresponding VSA questions. Repeating the test after one year with the same students increased retention of right answers by about 1.5 points. In summary, direct comparison questions in the VSA format are more difficult for our students to answer than questions in the SC format, conceivably because cueing and guessing are eliminated. Knowledge retention is present by repeating the very same examination format online. Retention of knowledge is higher when starting with VSA (group B) both for a subsequent SC format or a VSA format. These data would argue for more use of the VSA format at least in pharmacology examinations.

show abstract

“…45 Thus, although particular characteristics, such as exam length or test-taker cohort size, may influence the evaluation of individual assessments, item difficulty and discrimination are universally accepted as quality criteria in medical literature. [46][47][48] Nevertheless, alternative approaches, such as using open-book tests for formative assessment or employing different question formats (eg, essay, openended, uncued) have been recommended for use, especially in order to test higher-level problem-solving skills and provide effective discrimination among students. 49,50 However, it is still not entirely clear that these types of questions offer a better appraisal of higher-order cognitive functioning or contribute to assessments of higher validity.…”

Section: Discussionmentioning

confidence: 99%

Assessment Question Characteristics Predict Medical Student Performance in General Pathology

Hernandez

Magid

Polydorides

2021

Archives of Pathology &Amp; Laboratory Medicine

View full text Add to dashboard Cite

Context.— Evaluation of medical curricula includes appraisal of student assessments in order to encourage deeper learning approaches. General pathology is our institution's 4-week, first-year course covering universal disease concepts (inflammation, neoplasia, etc). Objective.— To compare types of assessment questions and determine which characteristics may predict student scores, degree of difficulty, and item discrimination. Design.— Item-level analysis was employed to categorize questions along the following variables: type (multiple choice question or matching answer), presence of clinical vignette (if so, whether simple or complex), presence of specimen image, information depth (simple recall or interpretation), knowledge density (first or second order), Bloom taxonomy level (1–3), and, for the final, subject familiarity (repeated concept and, if so, whether verbatim). Results.— Assessments comprised 3 quizzes and 1 final exam (total 125 questions), scored during a 3-year period (total 417 students) for a total 52 125 graded attempts. Overall, 44 890 attempts (86.1%) were correct. In multivariate analysis, question type emerged as the most significant predictor of student performance, degree of difficulty, and item discrimination, with multiple choice questions being significantly associated with lower mean scores (P = .004) and higher degree of difficulty (P = .02), but also, paradoxically, poorer discrimination (P = .002). The presence of a specimen image was significantly associated with better discrimination (P = .04), and questions requiring data interpretation (versus simple recall) were significantly associated with lower mean scores (P = .003) and a higher degree of difficulty (P = .046). Conclusions.— Assessments in medical education should comprise combinations of questions with various characteristics in order to encourage better student performance, but also obtain optimal degrees of difficulty and levels of item discrimination.

show abstract

Examinee Cohort Size and Item Analysis Guidelines for Health Professions Education Programs: A Monte Carlo Simulation Study

Cited by 9 publications

References 14 publications

Evaluation of the quality of multiple-choice questions according to the students' academic level

Evaluation of the quality of multiple-choice questions according to the students' academic level

Comparison of Online Tests of Very Short Answer versus Single Best Answers for Medical Students in a Pharmacology Course over One Year

Assessment Question Characteristics Predict Medical Student Performance in General Pathology

Contact Info

Product

Resources

About