Test-retest reliability of multiple true-false questions in preclinical medical subjects

Abstract

It has been suggested that the 'ideal' measure of reliability of an examination is obtained by test and retest using the one examination on the same group of students. However, because of practical and theoretical arguments, most reported reliabilities for multiple choice examinations in medicine are actually measures of internal consistency. While attempting to minimize the effects of potential interfering factors, we have undertaken a study of true test-retest reliability of multiple true-false type multiple choice questions in preclinical medical subjects. From three end-of-term examinations, 363 items (106 of 449 from term 1, 150 of 499 from term 2, and 107 of 492 from term 3) were repeated in the final examination (out of 999 total items). Between test and retest, there was little overall decrease in the percentage of items answered correctly and a decrease of only 3.4 in the percentage score after correction for guessing. However, there was an inverse relation between test-retest interval and decrease in performance. Between test and retest, performance decreased significantly on 33 items and increased significantly on 11 items. Test-retest correlation coefficients were 0.70 to 0.78 for items from the separate terms and 0.885 for all items that were retested. Thus, overall, these items had a very high degree of reliability, approximately the 0.9 which has been specified as the requirement for being able to distinguish between individuals.

Active learning on the ward: outcomes from a comparative trial with traditional methods

“…Students were advised not to answer a question if they were unsure of the answer. The reliability coefficient for this test can vary between 0.7 and 0.9 18,19 …”

“…The reliability coefficient for this test can vary between 0.7 and 0.9. 18,19 In the second phase, students participated in the academic ward rounds conducted by the staff member, who had been previously trained in the use of both learning methodologies. In the TM condition, the staff member read the medical chart with the patient history and established the diagnostic hypothesis and management plan, giving a short lecture at the bedside.…”

Increase in medical knowledge during the final year of undergraduate medical education in Germany

“…MCQ examinations can yield high reliability which is an important requirement to be able to distinguish between groups or individuals [8]. Furthermore, well-constructed MCQs can also assess higher-order cognitive processes such as interpretation and application of knowledge skills rather than just testing the recall of facts [9].…”

It takes only 100 true–false items to test medical students: true or false?

“…Multiple true/false (MTF) questions play a part in assessment in a number of medical courses (Schwartz et al, 1986;Richardson, 1992;Dixon, 1994). Their role has changed in many medical schools over the last decade, and MTF questions are now seldom used as the dominant assessment tool.…”