Measuring higher cognitive levels by multiple choice questions: a myth?

Ferland, Julie; Dorval, Jérémie; Levasseur, L.

doi:10.1111/j.1365-2923.1987.tb00675.x

Cited by 20 publications

(16 citation statements)

References 3 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…It is economical to administer and marking is standardized. Controversy continues as to whether the method assesses anything other than recall (Ferland et al 1987). It was consistently rated as the least desirable assessment method by the students in this study.…”

Section: Discussionmentioning

confidence: 99%

Formative assessment: a student perspective

1994

View full text Add to dashboard Cite

An educator's view would be that formative assessment has an important role in the learning process. This study was carried out to obtain a student perspective of the place of formative assessment in the curriculum. Final-year medical students at Royal Prince Alfred Hospital took part in four teaching sessions, each structured to integrate teaching with assessment. Three assessment methods were used; the group objective structured clinical examination (G-OSCE), structured short answer (SSA) questions and a pre/post-test multiple choice questionnaire (MCQ). Teaching sessions were conducted on the subject areas of traumatology, the 'acute abdomen', arterial disorders and cancer. Fifty-five students, representing 83% of those who took part in the programme, responded to a questionnaire where they were asked to rate (on a 5-point Likert scale) their response to general questions about formative assessment and 13 specific questions concerning the comparative value of the three assessment modalities. Eighty-nine per cent of respondents felt that formative assessment should be incorporated into the teaching process. The SSA assessment was regarded as the preferred modality to reinforce previous teaching and test problem-solving skills. The MCQ was the least favoured assessment method. The effect size variable between the total scores for the SSA and MCQ was 0.64. The variable between G-OSCE and SSA/MCQ was 0.26 and 0.33 respectively. Formative assessment is a potentially powerful method to direct learning behaviour. Students should have input into the methods used.

show abstract

Section: Discussionmentioning

confidence: 99%

Formative assessment: a student perspective

1994

View full text Add to dashboard Cite

show abstract

“…The MCQ can be modified for problem‐solving, but alternative solutions are still supplied rather than encouraging the real‐life situation of solving the problem in the individual's own way. Ferland et al (1987) found no difference in MCQs which set out to measure higher cognitive levels and suggested finding other instruments. The structured short‐answer question endeavours to do this.…”

Section: Mcq Methodsmentioning

confidence: 91%

Structured short-answer questions: an alternative examination method

Webber

1992

Medical Education

View full text Add to dashboard Cite

A structured short answer (SSA) examination has been developed for the MSc, Community Health in Developing Countries, to test competence in international public health. Questions can test knowledge or aim to assess problem-solving ability, but have the advantage that the candidates construct their own answer (rather than choosing from pre-formed answers in an MCQ), yet encourage sufficient precision for answers to be easily marked. The SSA more closely reflects participatory teaching and self-directed learning. Illustrated material is widely used.

show abstract

“…Comparisons of nonexperts to experts must account for the variability among experts in the approach to a specific problem. This creates the assessment problem seen with standardized patient examinations (SP), which have not captured increasing levels of expertise (Ferland et al 1987;Hodges et al 1999Hodges et al , 2002. One explanation is that the SP checklist, the specific items expected of the examinee, does not encompass the entirety of expert approaches.…”

Section: Cognitive Skill Developmentmentioning

confidence: 98%

“…A physician using non-expert reasoning can (and does) often make a correct diagnosis. Assessments that rely on a correct diagnosis, such as multiple-choice tests, poorly discriminate among different levels of expertise (Anderson 2004;Ferland et al 1987). Also, experts use problem-solving patterns that are similar to each other, but may not gather identical information.…”

Section: Cognitive Skill Developmentmentioning

confidence: 98%

“…Multiple-choice question tests and standardized patient examinations do not reliably measure clinical reasoning (Ferland et al 1987;Hodges et al 1999;Merrick et al 2002;Regehr et al 1998). Evaluation by supervising physicians forms the basis of assessment for advanced learners; however, physician observation of learners' information-gathering occurs infrequently (Nutter and Whitcomb 2001).…”

Section: Assessing Diagnostic Reasoningmentioning

confidence: 99%

See 1 more Smart Citation

Information-gathering patterns associated with higher rates of diagnostic error

Delzell

Chumley

Webb

et al. 2009

Adv in Health Sci Educ

View full text Add to dashboard Cite

Diagnostic errors are an important source of medical errors. Problematic information-gathering is a common cause of diagnostic errors among physicians and medical students. The objectives of this study were to (1) determine if medical students' information-gathering patterns formed clusters of similar strategies, and if so (2) to calculate the percentage of incorrect diagnoses in each cluster. A total of 141 2nd year medical students completed a computer case simulation. Each student's information-gathering pattern included the sequence of history, physical examination, and ancillary testing items chosen from a predefined list. We analyzed the patterns using an artificial neural network and compared percentages of incorrect diagnoses among clusters of information-gathering patterns. We input patterns into a 35 x 35 self organizing map. The network trained for 10,000 epochs. The number of students at each neuron formed a surface that was statistically smoothed into clusters. Each student was assigned to one cluster, the cluster that contributed the largest value to the smoothed function at the student's location in the grid. Seven clusters were identified. Percentage of incorrect diagnoses differed significantly among clusters (Range 0-42%, Chi (2) = 13.62, P = .034). Distance of each cluster from the worst performing cluster was used to rank clusters. This rank was compared to rank determined by percentage incorrect. We found a high positive correlation (Spearman Correlation = .893, P = .007). Clusters closest to the worst performing cluster had the highest percentages of incorrect diagnoses. Patterns of information-gathering were distinct and had different rates of diagnostic error.

show abstract

Measuring higher cognitive levels by multiple choice questions: a myth?

Cited by 20 publications

References 3 publications

Formative assessment: a student perspective

Formative assessment: a student perspective

Structured short-answer questions: an alternative examination method

Information-gathering patterns associated with higher rates of diagnostic error

Contact Info

Product

Resources

About