Multiple‐choice and Constructed Response Tests of Ability: Race‐based Subgroup Performance Differences on Alternative Paper‐and‐pencil Test Formats

Abstract

We present an example of an innovative constructed response test format–a write‐in/mark‐in paper‐and‐pencil test–as an alternative to the traditional multiple‐choice paper‐and‐pencil test, with the potential for reducing subgroup differences. We present subgroup differences data on these 2 paper‐and‐pencil test formats on an operational promotional exam in a sample of African American and White firefighters. The tests were designed to measure the same content domain. Using within‐subjects data that compared the performance of 13 African American and 14 White fire captains, and between‐subjects data that compared the performance of 21 African American and 49 White fire captains, several results were in the predicted direction such that subgroup differences were reduced on the constructed response test. However, these results did not reach statistical significance. Therefore, the study points to the need for additional research to further evaluate the promise of the constructed response test format.

Reviewing, Categorizing, and Analyzing the Literature on Black–White Mean Differences for Predictors of Job Performance: Verifying Some perceptions and Updating/Correcting Others

“…Further, we reported on several methods in this paper (e.g., work samples, assessment centers, biodata), but the methods could be parsed further by presentation mode (e.g., paper, video, or oral stimuli). Other dimensions to “method” include scoring, timing, response mode, and so forth (for examples of studies on open‐ended response mode, see Arthur, Edwards, & Barrett, 2002, or Edwards & Arthur, 2007).…”

The Three‐option Format for Knowledge and Ability Multiple‐choice Tests: A case for why it should be more commonly used in personnel testing

“…Similarly, Edwards and Arthur () and Arthur et al. () demonstrated that African American–White differences in test scores were smaller on a constructed‐response format compared to a MC format. (It is worth noting that these method comparisons were based on the same test content [Arthur & Villado, ].)…”

“…Numerous approaches have been investigated in efforts to reduce these observed differences (Arthur & Doverspike, ; Ployhart & Holtz, ). Of these, the method‐change approach (e.g., Arthur, Edwards, & Barrett, ; Chan & Schmitt, ; Edwards & Arthur, ) is the most relevant to the present study. In its most basic form, this approach focuses on the relationship between the test method and observed subgroup differences in test scores.…”

Does underrepresented=understudied? Recent research on racial and ethnic minorities in I-O psychology

“…For instance, Black-White differences on the Raven's Advanced Progressive Matrices failed to reach conventional levels of statistical significance only when perceived threat was low. Other testing research indicated that Black-White testing differences tend to be greater on more cognitively loaded competencies (Goldstein, Yusko, & Nicolopoulos, 2001) and multiple-choice as opposed to write-in response formats (Arthur, Edwards, & Barrett, 2002), and may be less influenced by differential dropout rates than previously believed (Tam, Murphy, & Lyall, 2004).…”