Evaluating the Quality of Higher Education Instructor-Constructed Multiple-Choice Tests: Impact on Student Grades

Abstract

Multiple-choice questions (MCQs) are commonly used in higher education assessment tasks because they can be easily and accurately scored, while giving good coverage of instructional content in a short time. However, studies that have evaluated the quality of MCQs used in higher education assessments have found many flawed items, resulting in misleading insights about student performance and contaminating important decisions. Thus, MCQs need to be evaluated statistically to ensure high-quality items are used as the basis of inference. This study evaluated the quality of 100 instructor-written MCQs used in an undergraduate midterm test (50 items) and final exam (50 items), making up 50% of the course grade, using the responses of 372 students enrolled in one first-year undergraduate general education course. Item difficulty, discrimination, and chance properties were determined using classical test theory and item response theory (IRT) statistical item analysis models. The two-parameter logistic (2PL) model consistently had the best fit to the data. The impact on overall course grades between the original raw score model and the IRT 2PL model showed 70% of students would receive the same grade (i.e., D to A), but only one-third would get the same mark using the standard augmented grade scale (i.e., A+ to D−). The analyses show that higher education institutions need to ensure MCQs are evaluated before student grading decisions are made.

Comparing Results of Alternate Format Test Questions to Standard Multiple Choice Test Questions in First, Second, and Third Year Baccalaureate Nursing Students

“…Classical Test Theory (CTT) proposes that multiple choice questions on any given examination should have a range of difficulty, with item difficulty determined by the proportion of test candidates answering the item correctly (called the P-value)[1] [17]. An exam question with a P-value between P > 0.2 and P < 0.8 is considered to be an acceptable test item, with values P < 0.2 being too difficult and > 0.8 being too easy [1]. Oermann and Gaberson [18] suggest the desired P-value for most tests should be in the 0.3 to 0.7 range.…”

“…The use of multiple choice questions on examinations is a common method of assessment in the health disciplines [1][2] [3][4] [5]. This format of questioning is frequently used as it can effectively and efficiently assess large numbers of students, be administered in a relatively short time period [1] [2], cover a broad range of subject matter, and be easily and objectively scored [2] [6]. In nursing education, use of multiple choice questions on examinations is often used in combination with other methods of evaluating student performance [7] [8].…”

Analisis Kualitas Butir Soal Tipe Hots Pada Kompetensi Sistem Rem Di Sekolah Menengah Kejuruan

“…Oleh karena itu dalam melakukan penilaian sangat dibutuhkan alat atau instrumen yang valid dan mampu mengukur kemampuan siswa secara objektif [1], [2]. Diantara berbagai macam alat penilaian yang bisa digunakan, soal pilihan ganda menjadi alat ukur yang paling banyak digunakan untuk penilaian siswa [3], [4]. Hal tersebut disebabkan oleh luasnya materi yang dapat dicakup dalam soal pilihan ganda dengan waktu penilaian yang relatif singkat.…”

“…Selanjutnya Brown & Abdulnabi (2017) melalui hasil penelitiannya menunjukkan bahwa penggunaan analisis item IRT memiliki dampak menguntungkan yang potensial pada keseluruhan nilai dan jumlah siswa yang lulus. Ini juga menunjukkan bahwa umpan balik sangat bermanfaat bagi guru sebagai acuan dalam penyusunan soal yang sesuai dengan karakteristik [4].…”

Empowerment of Female Students at Majmaah University, Saudi Arabia

“…Various studies in different course offerings of numerous educational institutions have been conducted for evaluating the quality of multiple-choice questions (MCQs) in examinations (Ramah et al 2020). Statistical approaches that commonly utilize test item parameters such as item difficulty, item discrimination, and distractor efficiency are regarded as sample dependent or based on the ability of the test takers or students (Brown and Abdulnabi 2017). Despite the presence of various literature on writing quality MCQs (Carriveau 2016;Bowkett and Walker 2018), there has been a paucity of studies that focused on the quality of MCQs based on how they were composed by instructors.…”