The Three‐option Format for Knowledge and Ability Multiple‐choice Tests: A case for why it should be more commonly used in personnel testing

Edwards, Bryan; Arthur, Winfred; Bruce, Leonardis L.

doi:10.1111/j.1468-2389.2012.00580.x

Cited by 10 publications

(5 citation statements)

References 61 publications

(130 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Later, this theory was supported by many other studies [ 10 , 31 ]. The principle of random guessing is applied if all the options have an equal probability of being chosen; however, tests are usually designed to meet the learning objectives of a taught curriculum, and test takers are expected to have full or partial knowledge of the test items, so they approach each option with some degree of knowledge, which is certainly not random guessing [ 46 ]. Examinees are familiar with the exam subject.…”

Section: Discussionmentioning

confidence: 99%

Decreasing the options’ number in multiple choice questions in the assessment of senior medical students and its effect on exam psychometrics and distractors’ function

Al-lawama

Kumwenda²

2023

BMC Med Educ

View full text Add to dashboard Cite

Background Studies that have investigated the effect options’ number in MCQ tests used in the assessments of senior medical students are scarce. This study aims to compare exam psychometrics between three- and five-option MCQ tests in final-year assessments. Methods A cluster randomized study was applied. Participants were classified into three groups, according to their academic levels. Students in each of those levels were randomized into either the three- or five-option test groups. Results Mean time to finish the five-option test was 45 min, versus 32 min for the three-option group. Cronbach’s alpha was 0.89 for the three-option group, versus 0.81 for the five-options, p-value = 0.19. The mean difficulty index for the three-option group was 0.75, compared to 0.73 for the five-option group, p-value = 0.57. The mean discriminating index was 0.53 for the three-option group, and 0.45 for the five-options, p-value = 0.07. The frequency of non-functioning distractors was higher in the five-option test, 111 (56%), versus 39 (39%) in the three-options, with p-value < 0.01. Conclusions This study has shown that three-option MCQs are comparable to five-option MCQs, in terms of exam psychometrics. Three-option MCQs are superior to five-option tests regarding distractors’ effectiveness and saving administrative time.

show abstract

Section: Discussionmentioning

confidence: 99%

Decreasing the options’ number in multiple choice questions in the assessment of senior medical students and its effect on exam psychometrics and distractors’ function

Al-lawama

Kumwenda²

2023

BMC Med Educ

View full text Add to dashboard Cite

show abstract

“…The results reported here for the rank-SJT must also be interpreted in the context of the fact that we used a five-option SJT. Hence, one can envisage a situation where four- or maybe even a three-option SJT (e.g., see Edwards, Arthur, & Bruce, 2012) might be considered to be less difficult and engender less information processing and cognitive demands than a five-response format. In addition, the rank-SJT may also be influenced by the structural dispersion or dissimilarity of the response options (i.e., the extent to which the relative effectiveness of the options are similar or dissimilar).…”

Section: Discussionmentioning

confidence: 99%

Comparative evaluation of three situational judgment test response formats in terms of construct-related validity, subgroup differences, and susceptibility to response distortion.

Arthur¹,

Glaze²,

Jarrett³

et al. 2014

Journal of Applied Psychology

Self Cite

View full text Add to dashboard Cite

As a testing method, the efficacy of situational judgment tests (SJTs) is a function of a number of design features. One such design feature is the response format. However, despite the considerable interest in SJT design features, there is little guidance in the extant literature as to which response format is superior or the conditions under which one might be preferable to others. Using an integrity-based SJT measure administered to 31,194 job applicants, we present a comparative evaluation of 3 response formats (rate, rank, and most/least) in terms of construct-related validity, subgroup differences, and score reliability. The results indicate that the rate-SJT displayed stronger correlations with the hypothesized personality traits; weaker correlations with general mental ability and, consequently, lower levels of subgroup differences; and higher levels of internal consistency reliability. A follow-up study with 492 college students (Study 2; details of which are presented in the online supplemental materials) also indicates that the rate response format displayed higher levels of internal consistency and retest reliability as well as favorable reactions from test takers. However, it displayed the strongest relationships with a measure of response distortion, suggesting that it is more susceptible to this threat. Although there were a few exceptions, the rank and most/least response formats were generally quite similar in terms of several of the study outcomes. The results suggest that in the context of SJTs designed to measure noncognitive constructs, the rate response format appears to be the superior, preferred response format, with its main drawback being that it is susceptible to response distortion, although not any more so than the rank response format.

show abstract

“…Multiple choice tests are often used to measure a person's cognitive abilities (Carretta & Ree, 2018;Edwards et al, 2012), both in formative assessment or summative assessment, as well as in school, college, or general such as civil servant selection. Siegfried and Wuttke (2019) also reported that the multiple-choice format is most widely used in college to measure students' cognitive performance.…”

Section: Introductionmentioning

confidence: 99%

Analysis of the distractor of the multiple-choice test using classical test theory (CTT) and item response theory (IRT)

Ihda,

Heri

2023

Materials of International Practical Internet Conference “Challenges of Science”

View full text Add to dashboard Cite

This study aimed to analyzed the functional distractor based on the results of the multiple-choice test using CTT and IRT. Data obtained mathematics test at one of the junior high schools (grade 7) in Sidoarjo in the 2021-2022 academic year. The test consists of 20 items, which are a collection of questions that have been standardized in the school curriculum. One hundred examinees attending this mathematics test. The analysis used Iteman 4.3 software for analysis distractor using CTT, and R program for analysis pseudo-guessing’s parameter using 3PL IRT model. The result showed that all items have distractors well function it (Prop. > 0,05) in CTT. Meanwhile, with IRT, 8 of the 20 questions were not good because the pseudo-guessing index was > 0.25. The result of this study provided important information for future study to examine the ability estimate when a test's fixed feature is the item-specific characteristic utilized for pseudo-guessing.

show abstract

The Three‐option Format for Knowledge and Ability Multiple‐choice Tests: A case for why it should be more commonly used in personnel testing

Cited by 10 publications

References 61 publications

Decreasing the options’ number in multiple choice questions in the assessment of senior medical students and its effect on exam psychometrics and distractors’ function

Decreasing the options’ number in multiple choice questions in the assessment of senior medical students and its effect on exam psychometrics and distractors’ function

Comparative evaluation of three situational judgment test response formats in terms of construct-related validity, subgroup differences, and susceptibility to response distortion.

Analysis of the distractor of the multiple-choice test using classical test theory (CTT) and item response theory (IRT)

Contact Info

Product

Resources

About