Web-Based Application to Eliminate Five Contraindicated Multiple-Choice Question Practices

Brunnquell, Andreas; Degirmenci, Ümüt; Kreil, Sebastian; Kornhuber, Johannes; Weih, Markus

doi:10.1177/0163278710370459

Cited by 7 publications

(10 citation statements)

References 11 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Many other common flaws could be detected through algorithmic means (Vasiliki et al, 2015). Within this study, we simply counted the number of characters within each option in order to identify the longest one, but others have used computational techniques to look for the most linguistically complex option instead (Brunnquell et al, 2011). Post-test statistical analysis of MCQs by established methods such as Classical Test Theory, would provide additional information regarding which items are poorly discriminating, and so potentially assist human evaluators in identifying item flaws.…”

Section: Discussionmentioning

confidence: 99%

The future of online testing and assessment: question quality in MOOCs

Costello

Holland

Kirwan

2018

Int J Educ Technol High Educ

View full text Add to dashboard Cite

If MOOCs are to play a role in the future of higher education it is imperative that we critically examine how they are currently functioning. In particular, questions persist about the role MOOCs will play in the future of formal accredited learning. As the focus turns from informal and free to formal, accredited and paid, greater scrutiny will be brought to bear on the quality of the courses themselves. Although there have been some empirical studies into the quality of MOOCs, a notable gap exists in that such research has not examined Multiple Choice Questions (MCQs) which are a key component of much MOOC assessment and testing. Previous research suggests that flawed MCQ items may compromise the reliability and validity of these assessments, potentially leading to inconsistent outcomes for students. This study was hence designed to examine MCQ quality in MOOCs. 204 MCQs were analysed, from a selection of 18 MOOCs, sampling the domains of computing, social science and health sciences. Over 50% of MCQs (112) contained at least one item flaw; 57 MCQs contained multiple flaws. A large proportion of MOOC MCQs violated itemwriting guidelines, which is comparable with previous studies examining the prevalence of flaws in assessments in more traditional educational contexts. The problem of low quality MCQs can be ameliorated by appropriate faculty training and pre-and post-test quality checks. These activities are essential if MOOCs are to become a force that can enable enhanced and improved pedagogies in the future of higher education, instead of simply proceeding to replicate existing poor practices at scale.

show abstract

Section: Discussionmentioning

confidence: 99%

The future of online testing and assessment: question quality in MOOCs

Costello

Holland

Kirwan

2018

Int J Educ Technol High Educ

View full text Add to dashboard Cite

show abstract

“…The list of question flaws was generated by a literature review (11)(12)(13)16), referencing a local institutional guideline for MCQ writing (18), review of the American Board of Radiology item writers' guide (19), and the National Board of Medical Examiners' item writing guide (20). All commonly cited independent ''flaws,'' which violate standard evidencebased principles of effective item writing, were included in the analysis.…”

Section: Mcq Flawsmentioning

confidence: 99%

“…Questions that violate widely agreed on rules are described in the education literature as flawed (9)(10)(11)(12)(13). In simple terms, a flawed question tends to test ''how good of a test taker'' someone is, rather than the relevant knowledge intended, which can disadvantage some students (10).…”

mentioning

confidence: 99%

Does Educator Training or Experience Affect the Quality of Multiple-Choice Questions?

2015

View full text Add to dashboard Cite

Rationale and Objectives: Physicians receive little training on proper multiple-choice question (MCQ) writing methods. Well-constructed MCQs follow rules, which ensure that a question tests what it is intended to test. Questions that break these are described as ''flawed.'' We examined whether the prevalence of flawed questions differed significantly between those with or without prior training in question writing and between those with different levels of educator experience. Materials and Methods:We assessed 200 unedited MCQs from a question bank for our senior medical student radiology elective: an equal number of questions (50) were written by faculty with previous training in MCQ writing, other faculty, residents, and medical students. Questions were scored independently by two readers for the presence of 11 distinct flaws described in the literature.Results: Questions written by faculty with MCQ writing training had significantly fewer errors: mean 0.4 errors per question compared to a mean of 1.5-1.7 errors per question for the other groups (P < .001). There were no significant differences in the total number of errors between the untrained faculty, residents, and students (P values .35-.91). Among trained faculty 17/50 questions (34%) were flawed, whereas other faculty wrote 38/50 (76%) flawed questions, residents 37/50 (74%), and students 44/50 (88%). Trained question writers' higher performance was mainly manifest in the reduced frequency of five specific errors.Conclusions: Faculty with training in effective MCQ writing made fewer errors in MCQ construction. Educator experience alone had no effect on the frequency of flaws; faculty without dedicated training, residents, and students performed similarly.Key Words: Multiple-choice questions; educator experience; question flaws; education. ªAUR, 2015P hysicians are rarely trained to properly write multiplechoice examinations, including those working in academic settings. However, this skill set has become much more relevant in recent years. With the transition to the new written format of radiology board certification examinations (1), the development of more rigorous self-assessment requirements for maintenance of certification examinations (2-4), and the greater inclusion of radiology into integrated medical student curricula (5), multiple choice radiology questions are in great demand.Well-constructed multiple-choice questions (MCQs) follow a set of parameters that ensure the question tests what it is intended to test (6-8). Questions that violate widely agreed on rules are described in the education literature as flawed (9-13). In simple terms, a flawed question tends to test ''how good of a test taker'' someone is, rather than the relevant knowledge intended, which can disadvantage some students (10). Previous literature examining MCQs has revealed that such mistakes are common within continuing medical education (CME) materials (14,15) and on health care sciences examinations (10,16).Previous authors have found that MCQ writing is improved after de...

show abstract

“…As a result demand for well-constructed MCQs has burgeoned. Studies have revealed, however, that examinations from medical school through CME are rife with questions that violate accepted standard question-writing principles [1][2][3][4][5][6]. In the medical education literature, the term for these nonstandard questions is "flawed" [1][2][3][7][8][9].…”

mentioning

confidence: 99%